Distributed ai-based image analysis system and method

Abstract

Provided are an image analysis system and method. The image analysis system comprises: a local device that generates, as one or more types of base data, detection results acquired for locally collected images by using an object detection model; and an inference device that includes one or more inference modules and receives the one or more types of base data from the local device, wherein the inference device distributes the base data to the one or more inference modules according to the type of base data, and collects, for each predetermined identifier, one or more inference results inferred from each of the inference modules to generate inference information for each identifier.

Description

Distributed AI-based image analysis system and method

[0001] Cross-citation with related applications

[0002] This application claims the benefit of priority based on Korean Patent Application No. 10-2024-0166774 filed November 20, 2024, and all contents disclosed in the document of said Korean patent application are incorporated herein as part of this specification.

[0003] The disclosure relates to an image analysis system and method, and more specifically, to a distributed artificial intelligence-based image analysis system and method.

[0004] Along with the recent advancements in artificial intelligence technology, various deep learning algorithms for image processing are being actively researched and developed. Most of these algorithms are designed based on Convolutional Neural Networks (CNNs). In particular, various inference algorithms such as object detection, semantic segmentation, pose estimation, and face recognition are being researched and developed. At the same time, recent advancements in hardware technology and AI semiconductor technology have brought about significant changes in the environments in which deep learning algorithms operate. For example, technology has advanced to the point where deep learning algorithms can run in real-time on small end devices, such as edge AI devices. To this end, various deep learning algorithms are being optimized and installed on edge devices, thereby inferring a large amount of information based on image data obtained from the real world.

[0005] However, most of these CNN deep learning algorithms belong to the field of supervised learning. Since they are learning-based algorithms, they have a limitation (scalability limitation) in that the model must be retrained or a new model created if the number of data classes to be obtained increases. For example, while a face recognition algorithm can recognize a human face, determining the facial posture requires integrating a pose estimation algorithm to retrain the model based on the training data, or implementing a logic structure that alternates between the two algorithms. Considering this in the case of humans, whenever various data needs to be extracted—such as whether a hat is worn, clothing style, mask usage, gender, age group, facial expression, seating position, height, hair color, glasses usage, companionship, skirt / pants wear, shoe type (dress shoes / sneakers / slippers), and long pants / shorts—all these cases must be learned. This inherently acts as a serious limitation in processing new data. Furthermore, even if a new model is created or classes are added to an existing model and executed alternately, inference speed slows down, memory usage increases, and overall performance degrades due to the increase in the number of classes. In particular, unlike high-performance supercomputers, this problem is even more severe in edge AI devices that lack performance. Furthermore, in terms of technological scalability, efficient and easy expansion is difficult because this problem occurs repeatedly whenever a new class is added. Above all, since numerous image and video deep learning models are based on vast amounts of training data, there is a need to find new approaches to problem-solving as well as improve performance.

[0006] The objective is to provide a distributed AI-based image analysis system and method that can solve the problem of having to retrain the model or create a new model when the number of data classes to be obtained increases.

[0007] An image analysis system according to one embodiment includes: a local device that generates detection results obtained using an object detection model for images collected locally as one or more types of base data; and an inference device that includes one or more inference modules and receives the one or more types of base data from the local device, wherein the inference device distributes the base data to the one or more inference modules according to the type of base data and aggregates one or more inference results inferred from each of the inference modules by a predetermined identifier to generate inference information by identifier.

[0008] The local device may perform object detection on one frame of the image to generate a first type of basis data, and store the generated first type of basis data in a local variable of the local device.

[0009] The local device may perform object detection on the frame to generate base data of a second type different from the first type, and store the generated base data of the second type in the local variable.

[0010] The local device may transmit one or more types of basis data stored in the local variable to the inference device when the data size of the local variable exceeds a predetermined reference size or reaches a predetermined transmission period.

[0011] The inference device may distribute the basis data of the first type to the first inference module among the one or more inference modules, and the first inference module may execute one or more inference operations to generate the inference result for each inference operation.

[0012] The inference device may distribute the basis data of the second type to the second inference module among the one or more inference modules, and the second inference module may execute one or more inference operations to generate the inference result for each inference operation.

[0013] The inference device may distribute the basis data of the second type to the second inference module and the third inference module among the one or more inference modules, and the second inference module may execute one or more inference operations to generate the inference result for each inference operation, and the third inference module may execute one or more inference operations to generate the inference result for each inference operation.

[0014] The above inference device can generate inference information for each identifier by configuring the inference result generated by the first inference module and the inference result generated by the second inference module to correspond to one of the identifiers.

[0015] The identifier-specific inference information comprises a first identifier-specific inference information generated in a first inference cycle among a predetermined inference cycle, a second identifier-specific inference information generated in a second inference cycle occurring after the first inference cycle among the predetermined inference cycles, and a third identifier-specific inference information generated in a third inference cycle occurring after the second inference cycle among the predetermined inference cycles, wherein the second identifier-specific inference information is connected to the first identifier-specific inference information and the third identifier-specific inference information is connected to the second identifier-specific inference information, forming a chain-type data structure, and the identifier-specific inference information may be stored in a storage medium of the inference device.

[0016] A pool is set for each of the first inference module and the second inference module to limit the maximum number of inference modules that can be executed simultaneously, and the inference device can distribute the basis data of the first type to the first inference module based on the maximum number set for the first inference module in the pool, and distribute the basis data of the second type to the second inference module based on the maximum number set for the second inference module in the pool.

[0017] The above-mentioned base data may include a set of multiple values ​​corresponding to each of a predetermined set of items.

[0018] The above system may further include a management device that receives status information regarding the object detection model from the local device and provides a new object detection model or update to the local device.

[0019] A video analysis method according to one embodiment is a video analysis method performed by a computing device comprising a processor, a storage medium, and a communication interface, wherein the processor receives from a local device, through the communication interface, one or more types of base data including detection results obtained using an object detection model for a video collected locally; the processor distributes the base data to one or more inference modules according to the type of the base data; and aggregates one or more inference results inferred from each of the inference modules according to a predetermined identifier to generate inference information for each identifier.

[0020] The step of distributing to one or more inference modules includes the step of the processor distributing a first type of basis data among the basis data to a first inference module among the one or more inference modules, and the method may further include the step of the processor causing the first inference module to execute one or more inference operations to generate the inference result for each inference operation.

[0021] The step of distributing to one or more inference modules further includes the step of the processor distributing base data of a second type different from the first type among the base data to a second inference module among the one or more inference modules, and the method may further include the step of the processor causing the second inference module to execute one or more inference operations to generate the inference result for each inference operation.

[0022] The step of distributing to one or more inference modules further includes the step of the processor distributing base data of a second type different from the first type among the base data to a second inference module and a third inference module among the one or more inference modules, and the method may further include the step of the processor causing the second inference module to execute one or more inference operations to generate the inference result for each inference operation, and causing the third inference module to execute one or more inference operations to generate the inference result for each inference operation.

[0023] The above method may further include the step of generating inference information for each identifier by configuring the processor such that the inference result generated by the first inference module and the inference result generated by the second inference module correspond to one of the identifiers.

[0024] The identifier-specific inference information may be stored in the storage medium in a chain-type data structure in which the identifier-specific inference information comprises: first identifier-specific inference information generated in a first inference cycle among predetermined inference cycles; second identifier-specific inference information generated in a second inference cycle occurring after the first inference cycle among predetermined inference cycles; and third identifier-specific inference information generated in a third inference cycle occurring after the second inference cycle among predetermined inference cycles, wherein the second identifier-specific inference information is connected to the first identifier-specific inference information and the third identifier-specific inference information is connected to the second identifier-specific inference information.

[0025] A pool is set for limiting the maximum number of concurrently executable inference modules for each of the first inference module and the second inference module, and the method may include the step of distributing the basis data of the first type to the first inference module based on the maximum number set for the first inference module in the pool; and the step of distributing the basis data of the second type to the second inference module based on the maximum number set for the second inference module in the pool.

[0026] The above-mentioned base data may include a set of multiple values ​​corresponding to each of a predetermined set of items.

[0027] According to the embodiments, the existing scalability limitation problem, which required continuous model training and updating, can be effectively resolved by processing final inference on a separate device, such as a server, when a class is added. Additionally, there is an added advantage in that concerns about technology theft are eliminated even if the edge AI device in the field is stolen or hacked, as the final inference logic does not reside on the device but only contains an object detection model that extracts underlying data. In particular, instead of the conventional method of directly using face images in a deep learning engine to infer information such as age or gender, a proprietary inference algorithm can be built using classical machine learning techniques (e.g., XGBoost) based on underlying data. Furthermore, as the processing process on the edge AI device is simplified into a continuous sequence of object detection, the processing time of the proprietary model is reduced, and consequently, Frames Per Second (FPS) performance can also be improved.

[0028] FIG. 1 is a drawing for explaining an image analysis system according to one embodiment.

[0029] FIG. 2 is a drawing for explaining an image analysis system and method according to one embodiment.

[0030] FIG. 3 is a drawing for explaining an image analysis system and method according to one embodiment.

[0031] FIG. 4 is a drawing for explaining an image analysis system and method according to one embodiment.

[0032] FIG. 5 is a drawing for explaining an image analysis system and method according to one embodiment.

[0033] FIG. 6 is a drawing for explaining an example of implementation of an image analysis system and method according to one embodiment.

[0034] FIG. 7 is a drawing for explaining an image analysis system and method according to one embodiment.

[0035] FIG. 8 is a drawing for explaining an example of implementation of an image analysis system and method according to one embodiment.

[0036] FIG. 9 is a drawing for explaining an image analysis system and method according to one embodiment.

[0037] FIG. 10 is a drawing for explaining an image analysis system and method according to one embodiment.

[0038] FIG. 11 is a drawing for explaining an image analysis system and method according to one embodiment.

[0039] FIG. 12 is a block diagram illustrating a computing device according to one embodiment.

[0040] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0041] Throughout the specification and claims, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another.

[0042] Terms such as "...part," "...unit," and "module" as described in the specification may refer to a unit capable of processing at least one function or operation described in this specification, and may be implemented as hardware or circuit, software, or a combination of hardware or circuit and software. Additionally, at least some components or functions of the distributed artificial intelligence-based image analysis system and method according to the embodiments described below may be implemented as a program or software, and the program or software may be stored on a computer-readable medium.

[0043] FIG. 1 is a drawing for explaining an image analysis system according to one embodiment.

[0044] Referring to FIG. 1, an image analysis system (1) according to one embodiment may include an inference device (10), a local device (20), and a management device (30). The inference device (10), the local device (20), and the management device (30) may exchange data with each other through a network (40).

[0045] The inference device (10) and the local device (20) may be computing devices. Specifically, the inference device (10) and the local device (20) may each include a processor, a storage medium, and a communication interface. Here, the term "storage medium" is used as a concept that comprehensively includes memory devices and storage devices (or storage). The inference device (10) and the local device (20) may each execute program code or instructions loaded into one or more memory devices through one or more processors. For example, the inference device (10) and the local device (20) may each be implemented as a computing device (50) as described below in relation to FIG. 12. In this case, one or more processors may correspond to the processor (510) of the computing device (50), and one or more memory devices may correspond to the memory (520) of the computing device (50). The program code or instructions may be executed by one or more processors to perform the functions of the inference device (10) and the local device (20) according to the embodiments described herein. In this specification, the term "module" is used to logically distinguish these functions performed by program code or instructions.

[0046] In some embodiments, the local device (20) may be placed locally where the images to be analyzed are collected, and the inference device (10) may be placed at a remote location. For example, the local device (20) may be installed and operated at a store (e.g., a cafe) where images are to be collected and analyzed, and the inference device (10) may be installed and operated at another location outside the store.

[0047] In some embodiments, the local device (20) may be implemented as an edge artificial intelligence device, and the inference device (10) may be implemented as a central server. The local device (20), which is an edge artificial intelligence device, is a device capable of processing data in real time at the site where the data is generated, and can collect data from peripheral devices such as cameras, sensors, and IoT (Internet of Things) devices, and perform preliminary analysis or inference thereon. The inference device (10) is implemented as a central server and can integrate and analyze data transmitted from the local device.

[0048] In some embodiments, the management device (30) may also be implemented as a computing device such as the inference device (10) described above, but sometimes the management device (30) may be implemented as a server on hardware integrated with the inference device (10).

[0049] The local device (20) can collect video locally. For example, the local device (20) can collect video captured inside the store using a device such as a camera installed in the store. The local device (20) can obtain detection results for the collected video using an object detection model. The object detection model may be an artificial intelligence algorithm that detects the location of a specific object in an image or video and classifies the object. The object detection model is based on a Convolutional Neural Network (CNN) and can perform all tasks of predicting the area where an object exists in an input image and determining the type of the object. The object detection model may include, for example, YOLO (You Only Look Once).

[0050] The local device (20) can generate detection results obtained using an object detection model on collected images as base data. Here, the base data refers only to the results of recognizing an object among the operations performed by the object detection model, and may refer to detection results for which no final inference has been performed.

[0051] For example, in the case of pose estimation, the baseline data may consist of 17 pivot points extracted corresponding to the major joints that make up the human body, and the final inference may involve performing additional tasks by tracking the 17 pivot points obtained per frame. An example of these additional tasks is obtaining the 3D node coordinates of the detected object by connecting the pivot points using Intersection over Union (IOU) and restoring them into 3D coordinates by mapping coordinate information obtained from two or more cameras into 3D. Another example of these additional tasks is generating a coordinate trajectory by connecting the xy pivot points obtained from pose estimation, and determining the behavior of the detected object by comparing this with previously learned behavior patterns.

[0052] As another example, in the case of human body recognition, the underlying data may be bounding box information (e.g., x1, y1, x2, y2 information), and the final inference may involve determining an identifier to distinguish a person and performing tracking through the connection of bounding boxes obtained for each frame. As an example of additional work, IOU and Kalman filtering may be performed, the height of the detected person may be inferred by considering the size and depth information (y2) of the bounding box, or the distance between the detected person and the table location may be calculated based on the x1, y1, x2, y2 information.

[0053] The local device (20) may generate the result of recognizing only the object as a 'recognition' process as base data, and may not directly perform the subsequent inference work by classifying it as an 'inference' process.

[0054] Specifically, the local device (20) can generate detection results obtained using an object detection model for collected images as one or more types of base data. For example, one or more types of base data may include face recognition base data obtained as a result of performing face recognition using the object detection model, human body recognition base data obtained as a result of performing human body recognition using the object detection model, and pose estimation base data obtained as a result of performing pose estimation using the object detection model.

[0055] The inference device (10) may receive one or more types of base data from the local device (20). Additionally, the inference device (10) may include one or more inference modules. Depending on the type of base data, the inference device (10) may distribute the base data received from the local device (20) to one or more inference modules. Here, the one or more inference modules may include, for example, an inference module that performs facial analysis of a detected person, an inference module that performs in-store tracking analysis of a detected person, an inference module that performs analysis of the seated position of a detected person in the store, and an inference module that analyzes the behavior or movement of a person in the store. In this case, the inference device (10) may distribute facial recognition base data among the base data received from the local device (20) to the inference module that performs facial analysis of a detected person, and distribute posture estimation base data among the base data received from the local device (20) to the inference module that analyzes the behavior or movement of a person in the store. Additionally, the inference device (10) can distribute human body recognition base data among the base data received from the local device (20) to an inference module that performs in-store tracking analysis of the detected person and an inference module that performs analysis of the seated position of the detected person in the store. One or more inference results inferred from each of these inference modules can be aggregated by a predetermined identifier (e.g., ID) to generate inference information by identifier. Accordingly, inference information for an anonymous person assigned an ID can be aggregated, and inference information for another anonymous person assigned a different ID can be aggregated.

[0056] Accordingly, inference information based on identifiers can be utilized to track the dwell time, movement patterns, seating locations, or accompanying status of the same individual at the level of anonymous identifiers. This can be applied to store operations, such as managing congested areas, controlling seat turnover, and providing customized promotions by time of day. Specifically, by tracking specific customers within the store at the "ID" level, it is possible to identify the movement patterns of the same person as they move between various areas. For instance, the entire process—from a customer entering, placing an order at the counter, moving to a seating area to stay for a certain period, and then heading toward the restroom or exit—can be recorded based on individual customer IDs. This serves as a foundation for understanding the behavioral patterns of the same individual without exposing personal identity. Furthermore, analyzing customer dwell time and movement paths through same-person tracking can improve operational efficiency. For instance, if a specific group of customers repeatedly flocks to a particular area, staffing can be prioritized in that area or table spacing can be adjusted to reduce congestion. Additionally, analyzing the average time customers spend in the store can be directly utilized for managing seat turnover or developing strategies for staffing during peak hours. Moreover, anonymous same-person tracking data can be linked to personalized customer services. For example, if the seating patterns or dwell times of the same individual are identified during each visit, sales can be increased by designating those areas as reservable zones or displaying customized promotions during specific times when dwell times are longer. Since no personal customer information is utilized in this process, it is possible to protect privacy while simultaneously achieving practical store management and marketing effects.

[0057] According to the present embodiment, when a local device (20) transmits base data corresponding only to the 'recognition' process among the 'recognition' and 'inference' processes to an inference device (10), the inference device (10) can perform an inference task corresponding to the 'inference' process. By adopting such a distributed structure, when the number of data classes to be finally obtained increases, the existing scalability limitation problem can be solved by processing the changed parts in the inference device (10) without the need to retrain the object detection model mounted on the local device (20) or create a new object detection model.

[0058] FIG. 2 is a drawing for explaining an image analysis system and method according to one embodiment.

[0059] Referring to FIG. 2, in relation to an image analysis system and method according to one embodiment, at step (S201), the local device (20) may load an object detection model, for example, into memory. Although the drawing depicts step (S201) as being performed prior to step (S202), the scope of the present invention is not limited to what is depicted, and step (S201) may be performed, for example, after step (S208) and before step (S210), or at other times within a scope that does not deviate from the purpose of implementation.

[0060] In step (S202), the local device (20) may generate information regarding the status of the object detection models. For example, the local device (20) may generate status information including values ​​such as a list of names of the object detection models it possesses or the versions of the object detection models it possesses. Then, in step (S203), the local device (20) may transmit the status information to the management device (30). In addition, in some embodiments, the local device (20) may request a latest version update for the object detection models it possesses.

[0061] In step (S204), the management device (30) identifies the local device (20) and can verify the object detection model specifications to be applied to the identified local device (20). Based on the object detection model specifications, the management device (30) can compare the status information received from the local device (20) to determine whether it is necessary to transmit a new object detection model or an update to the local device (20). If it is determined that it is necessary to transmit a new object detection model to the local device (20), in step (S205), the management device (30) can transmit the new object detection model to the local device (20), and in step (S206), the local device (20) can install the new object detection model received from the management device (30). Meanwhile, if it is determined that it is necessary to transmit an update to the local device (20), in step (S207), the management device (30) can transmit the update to the local device (20), and in step (S208), the local device (20) can install the update received from the management device (30). Subsequently, in step (S209), the management device (30) can update the management history of the object detection model for each local device.

[0062] Subsequently, in step (S210), the local device (20) can set an execution routine. For example, the local device (20) can set various options, such as the order in which object recognition is performed to generate one or more basis data, and conditions and periods for transmitting the basis data to the inference device (10). Once the execution routine is set, in step (S211), the local device (20) can perform detection according to the execution routine to generate basis data.

[0063] FIG. 3 is a drawing for explaining an image analysis system and method according to one embodiment.

[0064] Referring to FIG. 3, in relation to an image analysis system and method according to one embodiment, in step (S301), a local device (20) selects a frame from a local image, and in step (S302), for the selected frame, a detection result based on an object detection model can be generated as base data. Subsequently, in step (S303), the local device (20) can transmit the base data to an inference device (10).

[0065] In step (S304), the inference device (10) can distribute the base data to one or more inference modules according to the type of base data. Subsequently, in step (S305), the inference device (10) can generate inference information by identifier by configuring the inference results generated from one or more inference modules into identifiers.

[0066] Meanwhile, in step (S306), the local device (20) can generate a detection result based on an object detection model as base data for another frame. Here, the other frame may be a frame after the frame selected in step (S302). Additionally, in step (S307), the local device (20) can generate a detection result based on an object detection model as base data for yet another frame. Here, the other frame may be a frame after the frame selected in step (S306). Next, in step (S308), the local device (20) can perform compression and encryption on the base data generated in steps (S306) through (S307). Subsequently, in step (S309), the local device (20) can transmit the base data, after compression and encryption are completed, to the inference device (10). That is, the local device (20) can repeat the task of generating base data frame by frame from the local image, and when a certain amount of repetition is completed, it can collect the generated base data and provide it to the inference device (10) at once. In step (S310), the inference device (10) can perform decoding and decompression on the collected and received data, and distribute the base data to one or more inference modules according to the type of base data. Subsequently, in step (S311), the inference device (10) can generate inference information by identifier by configuring the inference results generated from one or more inference modules into identifiers.

[0067] According to some embodiments, the local device (20) may select frames only when the amount of change in movement of an object in the image exceeds a certain threshold, rather than simply based on consecutive frames when selecting frames to generate base data. For example, instead of processing every frame in an image where the background is nearly static, base data may be generated only for frames where the rate of change in the object's position coordinates has increased by a predetermined rate (e.g., 5%) or more, thereby reducing unnecessary data transmission.

[0068] Additionally, the local device (20) may add metadata in the form of tags, including the creation time per frame and the device ID of the local device, during the step (S308) of compressing and encrypting the base data. In this way, the inference device (10) can perform data integrity verification and device-specific data flow tracking simultaneously based on the metadata, rather than simply classifying data types during the step (S310) of decrypting and decompressing.

[0069] The inference device (10) distributes only the base data that has passed such integrity verification to the inference module, and the data that failed verification can be discarded or recorded in a separate security log. In this case, the security log can be linked with the management device (30) so that an update or inspection is automatically requested when errors occur repeatedly in a specific local device. Accordingly, in addition to the existing structure of bundling and transmitting base data, the frame selection criteria are limited based on the amount of object change, and an integrity tag is added to the transmitted data so that it can be processed after security verification on the inference device side, thereby simultaneously enhancing data efficiency and security.

[0070] FIG. 4 is a drawing for explaining an image analysis system and method according to one embodiment.

[0071] Referring to FIG. 4, in relation to an image analysis system and method according to one embodiment, in step (S401), a local device (20) selects one frame from a local image, and in step (S402), can perform object detection on the frame selected in step (S401) to generate first type of basis data. Subsequently, in step (S403), the local device (20) can store the generated first type of basis data in a local variable of the local device (20). For example, the local device (20) can perform object detection on one frame to generate face recognition basis data and store the generated face recognition basis data in a local variable.

[0072] Next, in step (S404), object detection can be performed on the frame selected in step (S401) to generate a second type of basis data different from the first type. Subsequently, in step (S405), the local device (20) can store the generated second type of basis data in a local variable of the local device (20). For example, the local device (20) can generate human body recognition basis data by performing object detection on the same frame from which the facial recognition basis data was generated, and store the generated human body recognition basis data in a local variable.

[0073] Next, in step (S406), object detection can be performed on the frame selected in step (S401) to generate a third type of basis data different from the first and second types. Subsequently, in step (S407), the local device (20) can store the generated third type of basis data in a local variable of the local device (20). For example, the local device (20) can generate pose estimation basis data by performing object detection on the same frame in which the face recognition basis data and human body recognition basis data were generated, and store the generated pose estimation basis data in a local variable.

[0074] In step (S408), the local device (20) can check whether a predetermined transmission condition is satisfied. Specifically, the local device (20) can determine that the transmission condition is satisfied if the data size of the local variable exceeds a predetermined reference size. Alternatively, the local device (20) can determine that the transmission condition is satisfied if a predetermined transmission cycle is reached. If it is determined that the transmission condition is satisfied, step (S409) is performed, and if it is determined that the transmission condition is not satisfied, step (S401) can be performed. That is, if it is determined that the transmission condition is not satisfied, the next frame in the local image is selected in step (S401), and steps (S402) through (S407) can be performed again for the selected frame.

[0075] In step (S409), the local device (20) can perform compression and encryption on the base data generated in steps (S402) through (S407) and stored in the local variable. Subsequently, in step (S410), the local device (20) can transmit the base data, which has been compressed and encrypted, to the inference device (10). In step (S411), the inference device (10) can perform decryption and decompression on the aggregated and received data, and distribute the base data to one or more inference modules according to the type of the base data.

[0076] In some embodiments, when the local device (20) sequentially generates base data of the first type, the second type, and the third type in the same frame, it may perform a procedure to verify the correlation between each base data. For example, if facial recognition base data is detected in the same frame but human body recognition base data is not detected, the local device (20) may mark the corresponding facial recognition base data as incomplete data, store it in a local variable, and exclude it from priority transmission even if transmission conditions are met.

[0077] Additionally, in the step (S408) of checking transmission conditions, the local device (20) may additionally consider whether the proportion of incomplete data within the local variable is above a certain threshold, in addition to simply the data size or transmission period. For example, if the incomplete data accounts for more than a predetermined proportion (e.g., 30%) of the total base data, the local device (20) may not immediately perform compression and encryption, but may delay transmission until supplementary data is generated by further analyzing the next frame.

[0078] Subsequently, when performing compression and encryption in step (S409), the local device (20) transmits the incomplete data with a separate tag attached, and the inference device (10) can identify the tag in the decryption step (S411) to process the incomplete data and the complete data separately. The inference device (10) distributes only the complete data to the inference module, and the incomplete data can be stored in a hold queue or reported to the management device (30) to request retransmission.

[0079] FIG. 5 is a drawing for explaining an image analysis system and method according to one embodiment.

[0080] Referring to FIG. 5, for a selected frame in a local image, a first type basis data (21), a second type basis data (22), and a third type basis data (23) may be generated, and these data may be provided to an inference device (10). The inference device (10) may include a first inference module (11), a second inference module (12), a third inference module (13), and a fourth inference module (14).

[0081] The inference device (10) distributes the first type base data (21) to the first inference module (11), and the first inference module (11) can execute one or more inference operations (IO1, IO2, IO3, IO4) to generate inference results (111, 112, 113, 114) for each inference operation (IO1, IO2, IO3, IO4). For example, the inference device (10) can distribute the first type base data (21), i.e., facial recognition base data, to the first inference module (11), i.e., an inference module that performs facial analysis of a detected person. The first inference module (11) can execute a first inference operation (IO1) that identifies the same person to generate a first inference result (111) including the same person identification result, and execute a second inference operation (IO2) that classifies facial expressions to generate a second inference result (112) including the facial expression classification result. Additionally, the first inference module (11) can execute a third inference operation (IO3) for recognizing gender to generate a third inference result (113) including a gender recognition result, and execute a fourth inference operation (IO4) for estimating age to generate a fourth inference result (114) including an age estimation result.

[0082] The inference device (10) may distribute the second type base data (22) to the second inference module (12) and the third inference module (13), and the second inference module (12) may execute one or more inference operations (IO5, IO6) to generate inference results (121, 122) for each inference operation (IO5, IO6), and the third inference module (13) may execute one or more inference operations (IO7, IO8) to generate inference results (131, 132) for each inference operation (IO7, IO8). For example, the inference device (10) may distribute the second type base data (22), i.e., human body recognition base data, to the second inference module (12), i.e., an inference module that performs in-store tracking analysis of a detected person. The second inference module (12) can execute a fifth inference operation (IO5) that performs tracking to generate a fifth inference result (121) that includes tracking results, and execute a sixth inference operation (IO6) that estimates the time spent in the store to generate a sixth inference result (122) that includes estimation results. Meanwhile, the inference device (10) can distribute the second type base data (22), i.e., human body recognition base data, to the third inference module (13), i.e., an inference module that performs analysis of the seated position of a detected person in the store. The third inference module (13) can execute a seventh inference operation (IO7) that performs position matching to generate a seventh inference result (131) that includes position matching results, and execute an eighth inference operation (IO8) that estimates the time spent in the seated position to generate an eighth inference result (132) that includes estimation results.

[0083] The inference device (10) distributes the third type base data (23) to the fourth inference module (14), and the fourth inference module (14) can execute one or more inference operations (IO9, IO10) to generate inference results (141, 142) for each inference operation (IO9, IO10). For example, the inference device (10) can distribute the third type base data (23), i.e., posture estimation base data, to the fourth inference module (14), i.e., an inference module that analyzes the behavior or movement of a person in a store. The fourth inference module (14) can execute a ninth inference operation (IO9) that restores the x, y, and z coordinates of 17 nodes to generate a ninth inference result (141) including the restored result, and execute a tenth inference operation (IO10) that restores the behavior to generate a tenth inference result (142) including the restored result.

[0084] FIG. 6 is a drawing for explaining an example of implementation of an image analysis system and method according to one embodiment.

[0085] Referring to FIG. 6, the basis data may include a set of multiple values ​​corresponding to each of a predetermined set of items.

[0086] The first type of base data (21) corresponding to the facial recognition base data may be implemented to include values ​​regarding items such as screen coordinates, height (height), width, center, screen depth, number of bounding boxes, and recognition score, but this is an example of implementation to aid understanding and does not limit the scope of the present invention.

[0087] Next, the second type of base data (22) corresponding to the human body recognition base data may be implemented to include values ​​regarding items such as screen coordinates, height (face size), width, center, screen depth, number of bounding boxes, recognition score, and whether it is front or back, but this is an example of implementation to aid understanding and does not limit the scope of the present invention.

[0088] Next, the third type of basis data (23) corresponding to the pose estimation basis data may be implemented to include values ​​regarding items such as node coordinates, height (height), width, center, depth on the screen, number of bounding boxes, and recognition score, but this is an example of implementation to aid understanding and does not limit the scope of the present invention.

[0089] FIG. 7 is a drawing for explaining an image analysis system and method according to one embodiment.

[0090] Referring to FIG. 7, the inference device (10) can generate inference information by collecting one or more inference results inferred from each inference module according to a predetermined identifier. Accordingly, inference information for an anonymous person assigned an ID can be collected, and inference information for another anonymous person assigned a different ID can be collected.

[0091] For example, the inference device (10) can generate inference information (R1) for an anonymous person identified by an identifier 'ID1' by configuring the inference results (111_1, 111_1, 113_1, 114_1) generated through inference operations (IO1, IO2, IO3, IO4) by the first inference module (11), the inference results (121_1, 122_1) generated through inference operations (IO5, IO6) by the second inference module (12), and the inference results (131_1) generated through inference operations (IO7) by the third inference module (13) to correspond to one identifier 'ID1'.

[0092] Additionally, the inference device (10) can generate identifier-specific inference information (R2) by configuring the inference results (111_2, 111_2, 113_2, 114_2) generated through inference operations (IO1, IO2, IO3, IO4) by the first inference module (11), the inference results (121_2, 122_2) generated through inference operations (IO5, IO6) by the second inference module (12), and the inference results (131_2) generated through inference operations (IO7) by the third inference module (13) to correspond to one identifier 'ID2' for any other anonymous person identified by the identifier 'ID2'.

[0093] FIG. 8 is a drawing for explaining an example of implementation of an image analysis system and method according to one embodiment.

[0094] Referring to FIG. 8, the inference information (R1) for each identifier may be implemented to include values ​​regarding items such as inference personal ID, facial expression, companion, height, gender, age group, seating position, time of entry, time of exit, time of stay, and behavior, but this is an example of implementation to aid understanding and does not limit the scope of the present invention.

[0095] FIG. 9 is a drawing for explaining an image analysis system and method according to one embodiment.

[0096] Referring to FIG. 9, in relation to an image analysis system and method according to one embodiment, in step (S901), a local device (20) may select one frame from a local image and generate a detection result based on an object detection model for the selected frame as base data. Then, in step (S902), the local device (20) may transmit the base data to an inference device (10). Subsequently, in step (S903), the inference device (10) may distribute the base data to one or more inference modules according to the type of base data, and may generate inference information by first identifier by configuring the inference results generated from one or more inference modules as identifiers. Here, the inference information by first identifier may be generated during the first inference cycle among predetermined inference cycles.

[0097] In step (S904), the local device (20) can select another frame from the local image and generate a detection result based on an object detection model for the selected frame as base data. Then, in step (S905), the local device (20) can transmit the base data to the inference device (10). Subsequently, in step (S906), the inference device (10) can distribute the base data to one or more inference modules according to the type of base data, and generate inference information by a second identifier by configuring the inference results generated from one or more inference modules as identifiers. Here, the inference information by a second identifier may be generated in a second inference cycle that occurs after the first inference cycle among the predetermined inference cycles.

[0098] In step (S907), the inference device (10) can connect the inference information for the second identifier to the inference information for the first identifier. That is, the inference device (10) can store the inference information for the second identifier in a storage medium so that the inference information for the first identifier is fully preserved by forming a data structure connected in a chain form without overwriting the inference information for the second identifier to the inference information for the first identifier.

[0099] Subsequently, in step (S908), the local device (20) may select another frame from the local image and generate a detection result based on an object detection model for the selected frame as base data. Then, in step (S909), the local device (20) may transmit the base data to the inference device (10). Subsequently, in step (S910), the inference device (10) may distribute the base data to one or more inference modules according to the type of base data, and generate inference information by a third identifier by configuring the inference results generated from one or more inference modules as identifiers. Here, the inference information by a third identifier may be generated in a third inference cycle that occurs after the second inference cycle among the predetermined inference cycles.

[0100] In step (S911), the inference device (10) can connect the inference information for the third identifier to the inference information for the second identifier. That is, the inference device (10) can store the inference information for the third identifier in a storage medium so that the inference information for the first identifier and the inference information for the second identifier are fully preserved by forming a data structure connected in a chain form without overwriting the inference information for the third identifier or the inference information for the second identifier.

[0101] In some embodiments, according to the embodiment, when the inference device (10) links inference information for a first identifier, a second identifier, and a third identifier in a chain form, it may not merely link them in chronological order but may also record quality indicators for each inference cycle. For example, the inference device (10) may store together the confidence score of the inference result generated in the first inference cycle (e.g., a range from 0 to 1) and the completeness of the input base data (e.g., whether all types of base data are included). Subsequently, when the inference information for a second identifier generated in the second inference cycle is linked to the inference information for a first identifier, the inference device (10) may insert a quality change event tag on the chain if the difference in confidence scores between the two cycles is greater than a certain standard. For example, if the confidence score was high because face recognition base data was normally included in the first cycle, but the confidence dropped sharply because the face area was obscured in the second cycle, an event tag called "confidence drop" may be recorded together in the chain structure.

[0102] Additionally, when the inference information for the third identifier is linked to the inference information for the second identifier, the inference device (10) may indicate a chain branch if the main attributes of the object to be inferred have changed by more than a certain range compared to the previous cycle (e.g., key, position, posture, etc.). In this way, even if there is a large change in the data flow per cycle, the subsequently branched chain can be tracked separately.

[0103] Accordingly, instead of simply chaining periodic inference information in chronological order, quality indicators and change events can be recorded together or managed in a branched structure, thereby further enhancing the reliability of inference results and the usefulness of analysis.

[0104] FIG. 10 is a drawing for explaining an image analysis system and method according to one embodiment.

[0105] Referring to FIG. 10, the inference device (10) can connect second identifier-specific inference information (R2) to first identifier-specific inference information (R1) for any anonymous person identified by an identifier called 'ID1'. Here, the first identifier-specific inference information is generated in the first inference cycle (T1) among predetermined inference cycles, and the second identifier-specific inference information may be generated in the second inference cycle (T2) that occurs after the first inference cycle (T1) among predetermined inference cycles. Accordingly, for any anonymous person, the first identifier-specific inference information (R1) and the second identifier-specific inference information (R2) can be fully preserved.

[0106] Additionally, the inference device (10) can link the inference information (R3) by the third identifier to the inference information (R3) by the second identifier for the same anonymous person identified by the identifier 'ID1'. Here, the inference information by the third identifier may be generated in the third inference cycle (T3) that occurs after the second inference cycle (T2) among the predetermined inference cycles. Accordingly, the inference information by the first identifier (R1), the inference information by the second identifier (R2), and the inference information by the third identifier (R3) can be fully preserved for any anonymous person.

[0107] FIG. 11 is a drawing for explaining an image analysis system and method according to one embodiment.

[0108] Referring to FIG. 11, a pool may be set up to limit the maximum number of inference modules that can be executed simultaneously for each inference module. The inference device (10) may create instances of inference modules based on each maximum number set for each inference module in the pool, and distribute base data to instances of inference modules so that multiple inference modules of the same type can operate simultaneously.

[0109] Specifically, a pool may be set to limit the maximum number of concurrently executable inference modules for each of the first inference module (11) and the second inference module (12). Here, for clarity and convenience of explanation, it is assumed that the maximum number of concurrently executable first inference modules (11) is 2 and the maximum number of concurrently executable second inference modules (12) is 1, but in reality, the maximum number may have a larger value within the range permitted by computing resources.

[0110] The inference device (10) can generate a first inference module instance (11A) and a second inference module instance (11B) based on a maximum number of 2 set for the first inference module (11) in the pool. The inference device (10) can distribute the first type base data (21) to the first inference module instance (11A) and the second inference module instance (11B) so that two first inference modules (11) can operate simultaneously. Accordingly, inference operations (IO1, IO2, IO3, IO4) are performed on the first inference module instance (11A) to generate inference results (111A, 112A, 113A, 114A), and inference operations (IO1, IO2, IO3, IO4) are performed on the second inference module instance (11B) to generate inference results (111B, 112B, 113B, 114B).

[0111] Meanwhile, the inference device (10) can generate a second inference module instance (12A) based on a maximum number of 1 set for the second inference module (12) in the pool. The inference device (10) can distribute the second type base data (22) to the second inference module instance (12A) to enable one second inference module (12) to operate. Accordingly, inference operations (IO5, IO6) are performed on the second inference module instance (12A) to generate inference results (121A, 122A). In this way, the operation of the inference module can be varied and adjusted according to performance-oriented or efficiency-oriented considerations of computing resources.

[0112] FIG. 12 is a block diagram illustrating a computing device according to one embodiment.

[0113] Referring to FIG. 12, an image analysis system and method according to embodiments can be implemented using a computing device (50). This computing device (50) can be implemented as various types of electronic devices, servers or similar devices, and its functions can be implemented through a combination of software and hardware.

[0114] The computing device (50) may include at least one of a processor (501) communicating via a bus (509), a memory (502), a storage device (503), a display device (504), a network interface device (505) providing access to a network (40) for communication with other entities, and an input / output interface device (506) providing a user input interface or a user output interface. Of course, the computer device (50) may additionally include any electronic device necessary to implement the technical concept described in this specification, although not shown in FIG. 12.

[0115] The processor (501) can be implemented as various types of computing devices, such as an MCU (Micro Controller Unit), AP (Application Processor), CPU (Central Processing Unit), GPU (Graphic Processing Unit), NPU (Neural Processing Unit), QPU (Quantum Processing Unit), etc. The processor (501) is a semiconductor device that executes instructions stored in memory (502) or storage device (503) and can perform a core role in the system. Program code and data stored in memory (502) or storage device (503) instruct the processor (501) to perform specific tasks, thereby enabling the operation of the entire system. The processor (501) can be configured to implement the functions or methods described above in relation to FIGS. 1 to 11.

[0116] The memory (502) and storage device (503) may include various forms of volatile or non-volatile storage media for storing and accessing data of the system. For example, the memory (502) may include read-only memory (ROM) or random access memory (RAM). In some embodiments, the memory (502) may be embedded inside the processor (501), in which case the data transfer speed between the memory (502) and the processor (501) may be very fast. In some other embodiments, the memory (502) may be located outside the processor (501), in which case the memory (502) may be connected to the processor (501) through various data buses or interfaces. Such connection may be made through various known means, for example, a PCIe (Peripheral Component Interconnect Express) interface for high-speed data transfer or a memory controller. Meanwhile, examples of storage devices (503) include HDD (Hard Disk Drive) or SSD (Solid State Drive), and the scope of the present invention is not limited to the elements listed above for the purpose of explanation.

[0117] In some embodiments, at least some configurations or functions of the image analysis system and method according to the embodiments may be implemented as a program or software executed on a computing device (50), and the program or software may be stored on a computer-readable recording medium or storage medium. Specifically, a computer-readable recording medium or storage medium according to one embodiment may have a program recorded on it for executing steps included in the implementation of the image analysis system and method according to the embodiments on a computer including a processor (501) that executes a program or instructions stored in a memory (502) or a storage device (503).

[0118] In some embodiments, at least some configurations or functions of the image analysis system and method according to the embodiments may be implemented using hardware or circuits of the computing device (50), or may be implemented using separate hardware or circuits that can be electrically connected to the computing device (50).

[0119] According to the embodiments described so far, the existing scalability limitations—where models had to be continuously trained and updated—can be effectively resolved by processing final inference on a separate device, such as a server, when a class is added. Additionally, there is an added advantage in that concerns regarding technology theft are eliminated even if the edge AI device in the field is stolen or hacked, as the final inference logic does not reside on the device but only contains an object detection model that extracts underlying data. In particular, instead of the conventional method of directly using face images in a deep learning engine to infer information such as age or gender, a proprietary inference algorithm can be built using classical machine learning techniques (e.g., XGBoost) based on underlying data. Furthermore, as the processing process on the edge AI device is simplified into a continuous sequence of object detection, the processing time of the proprietary model is reduced, and consequently, Frames Per Second (FPS) performance can also be improved.

[0120] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art to which the present invention belongs, using the basic concept of the present invention as defined in the following claims, also fall within the scope of the present invention.

Claims

1. A local device that generates detection results obtained using an object detection model on images collected locally as one or more types of basis data; and It includes one or more inference modules and an inference device that receives one or more types of base data from the local device, The above inference device is, Distribute the basis data to the one or more inference modules according to the type of the basis data, and One or more inference results inferred from each of the above-mentioned inference modules are aggregated by predetermined identifiers to generate inference information by identifier. Video analysis system.

2. In Paragraph 1, The above local device is, The object detection is performed on one frame of the above video to generate first type of basis data, and An image analysis system that stores the generated first type of basis data in a local variable of the local device.

3. In Paragraph 2, The above local device is, The object detection is performed on the above frame to generate base data of a second type different from the first type, and An image analysis system that stores the generated basis data of the second type in the local variable.

4. In Paragraph 3, The above local device is, An image analysis system that transmits one or more types of basis data stored in the local variable to the inference device when the data size of the local variable exceeds a predetermined reference size or reaches a predetermined transmission period.

5. In Paragraph 3, The above inference device is, The above-mentioned first type of base data is distributed to the first inference module among the above-mentioned one or more inference modules, and An image analysis system in which the first inference module performs one or more inference operations and generates an inference result for each inference operation.

6. In Paragraph 5, The above inference device is, The above second type of base data is distributed to the second inference module among the one or more inference modules, and An image analysis system in which the second inference module performs one or more inference operations and generates an inference result for each inference operation.

7. In Paragraph 5, The above inference device is, The above second type of base data is distributed to the second inference module and the third inference module among the one or more inference modules, and The second inference module executes one or more inference operations and generates an inference result for each inference operation, and An image analysis system in which the third inference module performs one or more inference operations and generates an inference result for each inference operation.

8. In Paragraph 6, The above inference device is, An image analysis system that generates inference information for each identifier by configuring the inference result generated by the first inference module and the inference result generated by the second inference module to correspond to one of the identifiers.

9. In Paragraph 8, The above identifier-specific inference information includes first identifier-specific inference information generated in a first inference cycle among a predetermined inference cycle, second identifier-specific inference information generated in a second inference cycle occurring after the first inference cycle among the predetermined inference cycles, and third identifier-specific inference information generated in a third inference cycle occurring after the second inference cycle among the predetermined inference cycles. An image analysis system in which the inference information for the second identifier is connected to the inference information for the first identifier and the inference information for the third identifier is connected to the inference information for the second identifier in a chain-type data structure, wherein the inference information for the identifier is stored in a storage medium of the inference device.

10. In Paragraph 6, A pool is set for each of the first inference module and the second inference module to limit the maximum number of inference modules that can be executed simultaneously, and The above inference device is, Based on the maximum number set for the first inference module in the above pool, the basis data of the first type is distributed to the first inference module, and An image analysis system that distributes base data of the second type to the second inference module based on the maximum number set for the second inference module in the pool above.

11. In Paragraph 1, The above-mentioned base data comprises a set of multiple values ​​corresponding to each of a predetermined set of items, in an image analysis system.

12. In Paragraph 1, An image analysis system further comprising a management device that receives status information regarding the object detection model from the local device and provides a new object detection model or update to the local device.

13. A method for image analysis performed by a computing device comprising a processor, a storage medium, and a communication interface, wherein The step of the processor receiving from a local device, via the communication interface, one or more types of base data including detection results obtained using an object detection model on images collected locally; The above processor distributes the basis data to one or more inference modules according to the type of the basis data; and A step comprising: aggregating one or more inference results inferred from each of the above-mentioned inference modules by predetermined identifiers to generate inference information by identifier Video analysis method.

14. In Paragraph 13, The step of distributing to one or more of the above inference modules is, The processor includes the step of distributing a first type of base data among the base data to a first inference module among the one or more inference modules, and The above method is, A video analysis method comprising further including the step of the processor causing the first inference module to execute one or more inference operations and generate an inference result for each inference operation.

15. In Paragraph 14, The step of distributing to one or more of the above inference modules is, The processor further includes the step of distributing base data of a second type, which is different from the first type among the base data, to a second inference module among the one or more inference modules, and The above method is, A video analysis method comprising the step of the processor causing the second inference module to execute one or more inference operations and generate an inference result for each inference operation.

16. In Paragraph 14, The step of distributing to one or more of the above inference modules is, The processor further includes the step of distributing base data of a second type, which is different from the first type among the base data, to a second inference module and a third inference module among the one or more inference modules, and The above method is, A video analysis method comprising further steps in which the processor causes the second inference module to execute one or more inference operations to generate the inference result for each inference operation, and causes the third inference module to execute one or more inference operations to generate the inference result for each inference operation.

17. In Paragraph 15, A video analysis method further comprising the step of generating inference information for each identifier by configuring the processor such that the inference result generated by the first inference module and the inference result generated by the second inference module correspond to one of the identifiers.

18. In Paragraph 17, The above identifier-specific inference information includes first identifier-specific inference information generated in a first inference cycle among a predetermined inference cycle, second identifier-specific inference information generated in a second inference cycle occurring after the first inference cycle among the predetermined inference cycles, and third identifier-specific inference information generated in a third inference cycle occurring after the second inference cycle among the predetermined inference cycles. A video analysis method in which the inference information for each identifier is connected to the inference information for each first identifier and the inference information for each third identifier is connected to the inference information for each second identifier in a chain-type data structure, wherein the inference information for each identifier is stored in the storage medium.

19. In Paragraph 15, A pool is set for each of the first inference module and the second inference module to limit the maximum number of inference modules that can be executed simultaneously, and The above method is, A step of distributing the basis data of the first type to the first inference module based on the maximum number set for the first inference module in the above pool; and A video analysis method comprising the step of distributing the basis data of the second type to the second inference module based on the maximum number set for the second inference module in the pool above.

20. In Paragraph 13, The above-mentioned base data comprises a set of multiple values ​​corresponding to each of a predetermined set of items, in an image analysis method.

Images