Image search and matching method using multi-model

Abstract

Disclosed are a method, an image management device, and a system for performing object re-identification. The image management device of the present specification receives feature information of an object from a camera device by an artificial intelligence model and stores same. The image management device, when the artificial intelligence model of the camera device is updated, receives and stores feature information of the object extracted by the updated model. The image management device may perform an object re-identification operation for an input query by generating query feature information by the model before update and query feature information by the model after update, respectively, and by comparing each piece of query feature information with the feature information corresponding to each model. Accordingly, even when an artificial intelligence model is updated, feature information extracted from an existing model may be utilized as is without the need to re-extract feature information, so that computational cost and time resources can be reduced.

Description

Image search and matching method using multiple models

[0001] This specification relates to an image management device and an object re-identification method using the same.

[0002] Recently, in video surveillance systems, object re-identification (Re-ID) technology is being actively researched to recognize objects such as people or vehicles from cameras and to determine whether they are the same object.

[0003] In conventional technology, video data captured by a camera is transmitted to a Video Management System (VMS), where an artificial intelligence model is applied to extract feature vectors, which are then stored in a database. Subsequently, when a user requests a search, a query image is received as input, the same AI model is applied to generate query feature vectors, and it is determined whether the objects are the same by comparing them with previously stored feature vectors.

[0004] As artificial intelligence models become more sophisticated, there are instances where existing models are updated to new ones. In such cases, the base feature vectors stored in the past are incompatible with the base feature vectors of the updated model, making comparison impossible even for identical objects. Furthermore, when updating a model, existing image data must be reloaded and re-extracted using the updated model; this requires massive computational resources and time, making it difficult to implement in real-time systems. Meanwhile, transmitting the original video directly from the camera increases network bandwidth and can even lead to privacy issues.

[0005] The purpose is to provide a method and device that can utilize existing feature information based on the first model without re-extracting it, even when an artificial intelligence model is updated from a first model to a second model to solve the aforementioned problem, and can reliably determine whether an object is the same even when the model version is updated.

[0006] The technical problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the following detailed description of the invention.

[0007] An image management device according to one embodiment of the present specification includes: a storage unit that receives and stores first feature information of an object generated by an artificial intelligence model from a camera device; and a processor that determines whether an object is the same by comparing an input query with the first feature information stored in the storage unit, wherein, when the artificial intelligence model is updated from a first model to a second model, additionally receives and stores second feature information generated from the updated model in the storage unit, and applies both the first model and the second model to the query to generate first and second query feature information corresponding to each model, and determines whether the object is the same by comparing it with the first and second feature information corresponding to each model stored in the storage unit.

[0008] The processor can provide a re-identification result of the object based on a similarity score of a first comparison result obtained by comparing the first feature information and the first query feature information, and a second comparison result obtained by comparing the second feature information and the second query feature information.

[0009] The camera device includes a first model and a second model, and the processor can receive the first feature information and the second feature information to which each model is applied and store them in the storage unit.

[0010] The camera device comprises a plurality of camera devices including either the first model or the second model, and the processor can receive feature information generated by each of the first and second camera devices and store it in the storage unit.

[0011] The above query may include video, audio, or text entered by the user.

[0012] If the above query is voice or text, the processor can generate query feature information from the query using a generative artificial intelligence model.

[0013] When the artificial intelligence model mounted on the camera device is updated from the first model to the second model, the first query feature information and the second query feature information may be generated by the first text encoder learned in conjunction with the first model and the second text encoder learned in conjunction with the second model for the query.

[0014] The above first model and second model can generate feature information of different dimensions.

[0015] The first model above can generate feature information based on shape information of the object, and the second model can generate the feature information based on visual attributes including color or pattern of the object.

[0016] The first model above includes either a convolutional neural network (CNN) or a transformer network, and the second model above may be either the convolutional neural network (CNN) or the transformer network.

[0017] The first model above may be a model trained for speed optimization, and the second model above may be a model trained for re-identification accuracy improvement.

[0018] The above processor can receive only the first feature information extracted from the image data, excluding the image data captured by the camera device.

[0019] The processor may determine whether the objects are the same by adopting only the comparison results among the first and second comparison results in which the reliability score is greater than or equal to a predetermined threshold, or by fusing the comparison results according to a predefined weight.

[0020] A method for performing object re-identification according to another embodiment of the present specification comprises: receiving and storing first feature information of an object generated by an artificial intelligence model from a camera device; determining whether an input query is the same object by comparing it with the stored first feature information; additionally receiving and storing second feature information generated from the updated model when the artificial intelligence model is updated from the first model to the second model; and applying both the first model and the second model to the query to generate first and second query feature information corresponding to each model, and determining whether the object is the same by comparing it with the stored first and second feature information, respectively.

[0021] The step of determining whether the above-mentioned object is the same may include a step of providing a re-identification result of an object based on a similarity score of a first comparison result obtained by comparing the first feature information and the first query feature information, and a second comparison result obtained by comparing the second feature information and the second query feature information.

[0022] The above camera device may further include a first model and a second model, and a step of receiving and storing first feature information and second feature information to which each model is applied.

[0023] The above camera device may further include a plurality of camera devices comprising either the first model or the second model, and a step of receiving and storing respective feature information generated by the first and second camera devices.

[0024] If the above query is an image, voice, or text input by a user, the method may further include the step of generating query feature information by a generative artificial intelligence model for the above query.

[0025] The first and second models above may be trained to be different from each other in one or more of the dimension of the feature vector, the type of feature extracted, the network structure, or the learning objective.

[0026] A system for performing object re-identification according to another embodiment of the present specification comprises: a camera device that applies an artificial intelligence model to a captured image to extract first feature information of an object; and an image management device that receives and stores the first feature information from the camera device, determines whether an input query is the same object by comparing it with the stored first feature information, and, when the artificial intelligence model is updated from the first model to the second model, receives and stores second feature information generated from the updated model from the camera device, applies both the first model and the second model to the query to generate first and second query feature information corresponding to each model, and determines whether the object is the same by comparing it with the stored first and second feature information, respectively.

[0027] When the artificial intelligence model of the camera device is updated from the first model to the second model, the artificial intelligence model of the image management device can also be synchronized and updated to the second model.

[0028] A method for performing object re-identification according to another embodiment of the present specification comprises: receiving an image captured from a camera device; applying a first model to the received image to generate and store first feature information; and comparing an input query with the stored first feature information to determine whether it is the same object; and, when the object identification model is updated from the first model to a second model, applying a second model to the received image to generate and store second feature information; and applying both the first model and the second model to the query to generate first and second query feature information corresponding to each model, and comparing them with the stored first and second feature information, respectively, to determine whether it is the same object.

[0029] When the object identification model is updated from the first model to the second model, the first feature information is maintained without being deleted or regenerated, and the first feature information can be used together with the second feature information to determine whether it is the same object.

[0030] According to one embodiment of the present specification, even if an artificial intelligence model is updated, feature information extracted from an existing model can be utilized as is without the need to re-extract it, thereby reducing computational costs and time resources.

[0031] In addition, according to one embodiment of the present specification, feature information by model version is managed together for an object, and by performing a comparison based on multiple models for a query, compatibility issues between model versions can be resolved and re-identification accuracy can be improved.

[0032] In addition, according to one embodiment of the present specification, by configuring the camera device to transmit only feature information instead of the original image, the amount of network transmission is reduced, and it is also advantageous for protecting personal information.

[0033] In addition, according to one embodiment of the present specification, since a plurality of camera devices and a plurality of model versions can be utilized, an object re-identification system with high scalability and flexibility can be implemented even in a large-scale surveillance environment.

[0034] In addition, according to one embodiment of the present specification, by combining voice and text-based queries with generative artificial intelligence, users can easily perform searches through various forms of input, thereby greatly improving user convenience.

[0035] The effects obtainable in this specification are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below.

[0036]

[0037] The accompanying drawings, included as part of the detailed description to aid in understanding the present specification, provide embodiments of the present specification and, together with the detailed description, also explain the technical features of the present specification.

[0038] FIG. 1 is a drawing for illustrating a surveillance camera system for implementing a method for performing object re-identification according to one embodiment of the present specification.

[0039] FIG. 2 is an exemplary block diagram of an image management device according to one embodiment of the present specification.

[0040] FIG. 3 is an exemplary drawing for explaining the concept of performing object re-identification in an image management device according to one embodiment of the present specification.

[0041] FIG. 4 is a flowchart of a method for performing object re-identification according to one embodiment of the present specification.

[0042] FIG. 5 is a diagram illustrating the operation of a system for performing object re-identification according to one embodiment of the present specification.

[0043] FIG. 6 is an exemplary diagram illustrating the comparison process between a text-based query and image feature information during model update according to one embodiment of the present specification.

[0044] FIG. 7 is a flowchart of a method for performing object recoloring according to another embodiment of the present specification.

[0045] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present specification, provide embodiments of the present invention and explain the technical features of the present specification together with the detailed description.

[0046]

[0047] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the invention.

[0048] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0049] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0050] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0051] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0052] FIG. 1 is a drawing for illustrating a surveillance camera system for implementing a method for performing object re-identification according to one embodiment of the present specification.

[0053] Referring to FIG. 1, a surveillance camera system (10) according to one embodiment may include an image processing device (100) and an external device (300). In one embodiment, the surveillance camera system (10) may further include a network server (200) capable of performing wired and wireless communication with the image processing device (100) and the external device (300).

[0054] The image processing device (100) may be an electronic device for capturing images and audio of a surveillance area by capturing a surveillance area. The image processing device (100) may capture a surveillance area in real time for surveillance or security purposes and may be provided with one or more devices. The image processing device (100) may be a camera placed at a fixed position in a specific location, a camera that can move automatically or manually along a certain path, or a camera that can be moved by a person or a robot, etc. The image processing device (100) may be a PTZ camera having pan, tilt, and zoom functions. The image processing device (100) may be an analog camera, or an IP camera or network camera used by connecting to a wired or wireless internet. Depending on the function and use of the camera, the image processing device (100) may have various shapes and sizes, such as dome type, box type, bullet type, camouflage type, etc.

[0055] In one embodiment, the image processing device (100) is a visible light camera and can generate a visible image according to the luminance distribution of an object by acquiring image information by detecting light. In one embodiment, the image processing device (100) is an infrared light camera (or thermal camera) and can generate a thermal image that displays different colors depending on the intensity by detecting radiant energy (thermal energy) emitted by an object in the form of an infrared wavelength, which is a type of electromagnetic wave, and measuring the intensity of the thermal energy.

[0056] The image processing device (100) may have the function of recording the surveillance area or taking a photograph. The image processing device (100) may have the function of recording sounds occurring in the surveillance area. The image processing device (100) may have the function of generating an alert or performing recording or taking a photograph when a change, such as movement or sound, occurs in the surveillance area.

[0057] The image processing device (100) can analyze the captured image itself and / or an image obtained by editing the image. For example, the image processing device (100) can detect objects in the image by using an object detection algorithm. The object detection algorithm may be an AI-based algorithm, and may detect objects by applying a pre-trained artificial neural network model.

[0058] The image processing device (100) can analyze an image to generate metadata and index information for said metadata. The image processing device (100) can analyze the image and / or audio included in the image together or separately to generate metadata and index information for said metadata.

[0059] The network server (200) can provide various functions to the image processing device (100) and the external device (300) through the network (400). In FIG. 1, the network server (200) is provided independently from the image processing device (100), but the embodiments of the present invention are not limited thereto. For example, the network server (200) may be embedded in the image processing device (100). A description of the network server (200) will be provided later.

[0060] The external device (300) may include a server (3001) and a client (3002).

[0061] The server (3001) may be a device that includes a DVR (Digital Video Recorder), NVR (Network Video Recorder), VMS (Video Management System), etc., and provides video storage, search, and management functions. The server (3001) may receive and store video and / or audio from the video processing device (100) and index the video and / or audio for user search. The server (3001) may analyze the video and / or audio received from the video processing device (100). In one embodiment, a plurality of servers (3001) built at physically different locations may be federated with each other and managed and controlled by a single software. In one embodiment, the server (3001) may be built in a government office, police station, hospital, central control center, central control center, comprehensive situation room, etc.

[0062] The client (3002) is a terminal including a desktop PC, tablet PC, slate PC, notebook computer, mobile device such as a smartphone, etc., and may include software and / or applications for performing video monitoring and search, etc. by a user (operator). The client (3002) may be connected to the image processing unit (100) and / or server (3001) via a network. In one embodiment, the client (3002) may manage and operate the image processing unit (100) and server (3001) based on the cloud.

[0063] In one embodiment, the client (3002) transmits an information provision request signal to the image processing device (100) or server (3001) requesting the provision of all or part of the image and / or audio, and may receive all or part of the image and / or audio from the image processing device (100) or server (3001). The client (3002) transmits an information provision request signal to the image processing device (100) or server (3001) requesting metadata and / or index information for the metadata obtained by analyzing the image and / or audio, and may receive metadata and / or index information from the image processing device (100) or server (3001).

[0064] The image processing device (100), network server (200), and external device (300) can perform wired and wireless communication through the network (400). The network (400) may include, for example, wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless internet such as 3G, 4G (LTE), 5G, WiFi, Wibro, Wimax, wireless LANs, CDMA, satellite communication, and wireless networks including short-range communication such as Bluetooth, RFID (Radio Frequency Identification), infrared communication (IrDA, infrared Data Association), UWB (Ultra Wideband), ZigBee, and NFC (Near Field Communication). In the case of wireless mobile communication, the network (400) may additionally include components such as a base station (BTS), a mobile switching center (MSC), a home location register (HLR), an access gateway that enables the transmission and reception of wireless packet data, and a Packet Data Serving Node (PDSN). The scope of this specification is not limited thereto.

[0065] Meanwhile, for the convenience of explanation, the image processing device (100) of FIG. 1 is referred to as a camera device (100).

[0066] FIG. 2 is an exemplary block diagram of an image management device according to one embodiment of the present specification. The image management device (20) shown in FIG. 2 may be an exemplary device (or server) capable of performing the same function as the server (3001) shown in FIG. 1.

[0067] Referring to FIG. 2, the image management device (20) may include a communication unit (21), an input unit (22), an interface (23), a display unit (24), an AI processor (25), a memory (26), and a database (27).

[0068] The image management device (20) can analyze metadata received from the image processing device (100, hereinafter referred to as the 'camera device') to extract feature information of objects included in the image. Then, it constructs a database that a user can search by comparing it with previously stored feature information of objects. To this end, the image management device (20) includes a processor (28), a memory (26), an input unit (22), and a display unit (24). These components can be connected to each other and communicate via a bus.

[0069] The communication unit (21) can receive video data, voice data, still images, and / or metadata from the camera device (100). According to one embodiment, the communication unit (21) can receive video data, voice data, still images, and / or metadata from the camera device (10) in real time. The communication interface can perform at least one communication function among wired / wireless LAN (Local Area Network), Wi-Fi, ZigBee, Bluetooth, and Near Field Communication.

[0070] Input units (22) include a mouse, keyboard, joystick, remote control, etc. These input units can be connected to a bus through an input interface (23) including a serial port, a parallel port, a game port, a USB, etc. However, if the image management device (200) provides a touch function, the display unit (24) may include a touch sensor. In this case, there is no need to provide a separate input unit (22), and the user can directly input a touch signal through the display unit (24).

[0071] The interface (23) may include a network interface, a video interface, an input interface, etc. The network interface may include a network interface card, a modem, etc.

[0072] Even if the image management device (20) provides a touch function, if the display unit (24) does not include a touch sensor, a separate touch pad may be provided as an input unit (22). Various types of displays may be used for the display unit (24), such as LCD (Liquid Crystal Display), OLED (Organic Liquid Crystal Display), CRT (Cathode Ray Tube), and PDP (Plasma Display Panel). This display unit (24) is connected to a bus through a video interface (not shown), and data transmission between the display unit (24) and the bus can be controlled by a graphics controller (132).

[0073] The AI ​​processor (25) is for artificial intelligence image processing and applies a deep learning-based object detection algorithm learned as an object of interest in an image acquired through a surveillance camera system according to one embodiment of the present specification. The AI ​​processor (25) may be implemented as a single module with a processor (26) that controls the entire system, or as an independent module. In one embodiment of the present specification, the YOLO (You Only Lock Once) algorithm may be applied for object detection. YOLO is an AI algorithm suitable for surveillance cameras that process real-time video because it has a fast object detection speed. Unlike other object-based algorithms (Faster R-CNN, R_FCN, FPN-FRCN, etc.), the YOLO algorithm resizes a single input image and passes it through a single neural network only once, resulting in the output of a bounding box indicating the location of each object and a classification probability of what the object is. Finally, a single object is detected once through non-max suppression.

[0074] Meanwhile, it should be noted that the object recognition algorithm disclosed in this specification is not limited to the aforementioned YOLO and can be implemented with various deep learning algorithms.

[0075] Meanwhile, the learning model for object recognition applied in this specification may include a neural network model trained to extract the gender of a person when the object included in the aforementioned image is a person. As training data for training the neural network model, multiple images with different shooting location information and shooting time information obtained from multiple surveillance cameras separated by a certain distance or more are defined as input data, and a network may be trained to be designed to extract gender and image feature information from the multiple images.

[0076] The memory (26) stores various object information, and a database (27) is constructed by the processor (28). The memory (26) includes a non-volatile memory device and a volatile memory device. The non-volatile memory device is preferably a NAND flash memory that is small in volume, lightweight, and resistant to external shocks, and the volatile memory device is preferably a DDR SDRAM.

[0077] The database (27) is a storage for systematically managing feature information of an object stored in the memory (26), and stores feature information of an object transmitted from a camera device and maintains an indexing structure so that it can be compared with query feature information corresponding to an input query. Through this, similarity calculation results, comparison scores, and metadata can be managed to efficiently determine whether the objects are the same.

[0078] In one embodiment, the database (27) may include a storage structure separated by model version. For example, by independently storing and managing feature information extracted from the first artificial intelligence model and feature information extracted from the second artificial intelligence model, the existing feature information can be utilized as is without re-extracting it even if the artificial intelligence model is updated.

[0079] Additionally, the database (27) may include a vector search engine for high-speed searching of large-scale feature information and may apply Approximate Nearest Neighbor (ANN) based indexing techniques (e.g., FAISS, Annoy, HNSW, etc.). This enables real-time object re-identification even in a large-scale surveillance network environment.

[0080] In addition, the database (27) also stores management information such as user accounts, query history, and access logs, which can be used for security management and performance optimization of the system.

[0081] All components included in the processor (28) may be connected to the bus through at least one interface or adapter, or directly connected to the bus. Additionally, the bus may be connected to other subsystems in addition to the components described above. The bus may include a memory bus, a memory controller, a peripheral bus, and a local bus.

[0082] The processor (28) controls the overall operation of the image management device (20). For example, when metadata is received from a camera device (100) or a separate engine, feature information of an object included in the image can be extracted from the metadata and stored in a database (27). As an example of the metadata, the present specification may include the gender of an object (person) included in the image, feature information of the image, location information where the image capturing device is located, shooting time information, etc.

[0083] The processor (28) is preferably a CPU (Central Processing Unit), MCU (Micro Controller Unit), or DSP (Digital Signal Processor), but is not limited thereto and various logic operation processors may be used.

[0084] According to one embodiment, the processor (28) is a central control unit that performs overall control of the image management device (20) and is connected to the communication unit (21), input unit (22), interface (23), display unit (24), AI processor (25), memory (26), and database (27) to control data transmission and reception and operation among them.

[0085] The processor (28) manages metadata and feature information received from the camera device through the communication unit (21), and can generate query feature information by calling an operation module stored in the AI ​​processor (25) or memory (26) for an input query. In addition, it performs an operation to determine whether the object is the same by comparing the feature information of an object stored in the database (27) with the query feature information.

[0086] In one embodiment, the processor (28) determines whether to update the artificial intelligence model, and if the first model is updated to the second model in the camera device, it can synchronize the internal operation logic of the image management device in response. For example, it can additionally receive feature information generated from the updated second model and store it in the database (27), generate query feature information by applying both the first model and the second model to the input query, and then determine whether they are the same object by comparing the corresponding feature information of each model.

[0087] Additionally, the processor (28) can derive a final re-identification result by applying a reliability score or weight fusion to multiple comparison results, and this result can be provided to the user through the display unit (24).

[0088] FIG. 3 is an exemplary drawing for explaining the concept of performing object re-identification in an image management device according to one embodiment of the present specification.

[0089] A camera device (100) captures a surveillance space to acquire image data (31). The acquired image data (31) may detect objects of interest (32), such as people, by an object detection module. The detected objects may be analyzed by a feature extractor (33) to generate feature vectors (34) that reflect the external attributes of the objects. The feature vectors (34) may be represented, for example, as numerical vectors of hundreds of dimensions.

[0090] The generated feature vector (34) can be transmitted to the image management device (20) and stored in the image management device (20). As described above, the image management device (20) may include a VMS, NVR, and cloud device. When a user requests a search for a specific object, a query (36) is entered.

[0091] The query (36) may be, for example, image data, and the image management device (200) may extract a feature vector for the query object and compare it with the feature vectors stored in the database (35). During the comparison process, the similarity between each feature vector is calculated, and the result may be provided as a search result (37) through a similarity search module. The search result (37) may include similarity scores such as 0.9, 0.8, 0.4, for example, and may present objects sorted in order of high similarity to the query object. Through this, the user can identify identical objects or similar objects.

[0092] In one embodiment of the present specification, the query may be generated not only by image data but also by still images, external image files uploaded by a user, or non-image data such as text and voice. In particular, text or voice may be converted into feature information reflecting the attributes of an object through a generative artificial intelligence model, and the user may request object re-identification with only the input of a simple description.

[0093] In addition, in one embodiment of the present specification, the image management device (20) may generate feature information for each image-based query using multiple artificial intelligence models, such as a convolutional neural network (CNN) or a Transformer network. The generated feature information may be fused and utilized for final comparison. Additionally, voice or text queries may be converted into embedding vectors and compared in the same latent space as image feature vectors.

[0094] According to one embodiment of the present specification, the image management device (20) can calculate the similarity not only directly between query feature information and feature information received from the camera device (100) and stored in a database, but also after performing filtering based on specific attributes (e.g., color, pattern, motion, etc.). Additionally, comparison results generated from a plurality of models may be fused based on weights or selectively adopted according to reliability criteria.

[0095] FIG. 4 is a flowchart of a method for performing object re-identification according to one embodiment of the present specification. The method disclosed in FIG. 4 can be implemented by a processor (28) of an image management device (20).

[0096] Referring to FIG. 4, the processor (28) receives first feature information of an object generated by a first artificial intelligence model from the camera device (100) (S400). The first feature information may include various attributes such as the appearance, color, and pattern of an object extracted from an image captured by the camera device (100). The image management device (20) may receive the first feature information through a communication unit and store it in a storage unit.

[0097] The first artificial intelligence model may be based on a convolutional neural network (CNN). A CNN can hierarchically extract features from image data ranging from low-level features (edges, colors, textures) to high-level features (shape, contours, patterns). According to one embodiment, a lightweight network such as ResNet, Inception, or MobileNet may be applied to the first artificial intelligence model, and a speed-optimized structure may be selected for real-time operation. The first artificial intelligence model may generate output in the form of a fixed dimension (feature embedding) in the form of a vector. For example, it may be expressed as 256 dimensions, 512 dimensions, etc., and may be trained so that the vector distance between identical objects is close and between different objects is far apart.

[0098] The processor (28) can determine whether the input query is the same object by performing a comparison with the first feature information (S410). Here, the query can be provided in various data formats such as video, voice, and text, and can be converted into query feature information through a generative artificial intelligence model if necessary.

[0099] The processor (28) can determine whether the artificial intelligence model of the camera device (100) has been updated (S420). If the processor (28) determines that the artificial intelligence model of the camera device (100) has been updated from the first model to the second model (S420: Y), it can receive the second feature information generated by the updated second model from the camera device (100) and store it in a storage unit (S430).

[0100] Accordingly, the image management device (20) can store both the first feature information based on the first model and the second feature information based on the second model.

[0101] According to one embodiment, when the camera device (100) transmits feature information of an object, it may transmit including metadata such as a model ID, version number, or hash value indicating which artificial intelligence model generated the feature information. In this case, the processor (28) can determine whether the artificial intelligence model used in the camera device (100) has been updated from a first model to a second model by analyzing the received metadata.

[0102] Additionally, according to one embodiment, when the model is updated by the camera device (100), it may transmit a control signal or an update notification message to the image management device (20) indicating the model update. Accordingly, the processor (28) can receive the update information and directly recognize that the model version has been changed.

[0103] In addition, according to one embodiment, if both the camera device (100) and the image management device (20) are managed in a central server or cloud environment, the central server can perform model distribution and updates in an integrated manner. In this case, the processor (28) can check whether the model is updated by periodically checking or synchronizing the version information of the currently active model from the central server. Furthermore, if the first model and the second model generate feature information of different dimensions or if the data structure of the extracted feature information is different, the processor (28) may indirectly check whether the model is updated by analyzing the format of the received feature information. The method by which the image management device (20) checks whether the artificial intelligence model stored in the camera device (100) is updated is not limited to the above-described example, and various methods may be applied.

[0104] The processor (28) can apply both the first model and the second model to the input query to generate the first query feature information and the second query feature information, respectively (S440).

[0105] As described above, the input query can be input as video, still image, text, or voice, and in the case of a text or voice query, it can be converted into feature information in the form of a vector reflecting the attributes of an object through a generative artificial intelligence model. According to one embodiment, text input can be vectorized (embedding) using an LLM (Transformer-family neural network).

[0106] According to one embodiment, when a video or image query is input, the processor (28) may generate first query feature information by applying a first model and simultaneously generate second query feature information by applying a second model. At this time, the first query feature information may be expressed as a vector that primarily emphasizes the shape, outline, and structural features of an object, and the second query feature information may be expressed as a vector that includes visual attributes such as color, pattern, and text.

[0107] Meanwhile, when the first model of the camera device (100) is updated to the second model, as described above, the first feature information based on the first model is already stored in the image management device (20), so the first model may still be required to compare queries. Therefore, even if the model included in the image management device (20) is updated from the first model to the second model, the first model may not be deleted but maintained, or if necessary, the first model may be implemented to be executed as a backward compatible module.

[0108] The processor (28) can determine whether the object is the same by comparing the first query feature information with the first feature information and comparing the second query feature information with the second feature information (S450).

[0109] According to one embodiment, the processor (28) must generate feature information of the same representation space by applying the same model to process the query. That is, since the first feature information is in the representation space created by the first model, the query must also generate first query feature information through the first model to enable comparison. Similarly, since the second feature information is in the representation space of the second model, the query must also generate second query feature information through the second model.

[0110] According to one embodiment, a plurality of comparison results may be fused based on a distance score or selectively utilized based on a reliability threshold.

[0111] Accordingly, the object re-identification method according to one embodiment of the present specification can accurately determine whether an object is the same by utilizing the previously stored first feature information as is without the need to re-extract it, even when the artificial intelligence model is updated, and by using it in parallel with the feature information generated by the updated second model.

[0112] FIG. 5 is a diagram illustrating the operation of a system for performing object re-identification according to one embodiment of the present specification.

[0113] The camera device (100) applies an artificial intelligence model to captured images (51, 52) to extract feature information of an object. Initially, a first model (M1) is applied to generate first feature information (53) of the object, which is then transmitted to the image management device (20). Subsequently, when the artificial intelligence model of the camera device (100) is updated from the first model (M1) to the second model (M2), the updated second model (M2) is applied to additionally generate second feature information (54) of the same object, which is then transmitted to the image management device (20). At this time, the image management device (20) receives the second feature information generated from the updated model and stores it in a storage unit.

[0114] Meanwhile, the image management device (20) applies both the first model (M1) and the second model (M2) to the input query (59) to generate the first query feature information (55) and the second query feature information (56), respectively.

[0115] The first query feature information (55) is compared with the first feature information (53), and the second query feature information (56) is compared with the second feature information (54), so that a comparison corresponding to each of the feature information before and after the model update can be made.

[0116] Subsequently, the image management device (20) can calculate a similarity (distance, 57, 58) based on the first comparison result of the first feature information (53) and the first query feature information (55) and the second comparison result of the second feature information (54) and the second query feature information (56). These comparison results are integrated to calculate a total distance, and finally, a re-identification result based on whether the objects are the same can be provided to the user.

[0117] Meanwhile, according to one embodiment, the object re-identification system may include a plurality of camera devices (100). Among the plurality of camera devices, the first camera device may apply a first model (M1) to extract first feature information included in the captured image. Additionally, the second camera device may apply a second model (M2) to extract second feature information of the same object included in the captured image. The first and second feature information may be transmitted to an image management device (20) after being extracted from each camera device (100). The image management device (20) receives the first and second feature information and stores it in a storage unit, and then applies both the second model and the second model to a query input thereafter to generate first query feature information and second query feature information, respectively. The image management device (20) compares the first feature information and the second query feature information stored in the storage unit, and determines whether they are the same object by comparing the stored second feature information and the second query feature information. In other words, even if multiple camera devices are equipped with different artificial intelligence models, the image management device can integrate and utilize heterogeneous feature information, thereby improving the accuracy and flexibility of object re-identification.

[0118] In addition, according to one embodiment, the case in which the first model is updated to the second model in the camera device (100) has been described, but the feature information extracted by each model may be feature information that is not compatible with each other. For example, the first model and the second model may differ in the dimension of the generated feature information. The first model generates a feature vector of relatively low dimension (e.g., 512 dimensions) to improve computation speed, whereas the second model generates a feature vector of high dimension (e.g., 1024 dimensions) to provide more precise identification information.

[0119] Additionally, according to one embodiment, the first model and the second model may be distinguished based on the difference in the object attributes they emphasize. For example, the first model generates feature information primarily based on the external shape information of the object (e.g., body proportions, contours, silhouette, etc.), whereas the second model generates feature information based on visual attributes such as the color, pattern, and texture of the object, thereby reflecting various perspectives of the same object.

[0120] Additionally, according to one embodiment, the first model can extract feature information based on a convolutional neural network (CNN)-based network structure, and the second model can extract feature information based on a transformer network structure. In this case, the CNN-based model has strengths in extracting local patterns and is fast, whereas the transformer-based model can reflect long-range dependencies and is advantageous for precise identification.

[0121] In addition, according to one embodiment, the first model and the second model may be distinguished by a difference in the purpose of learning. For example, the first model is trained for speed optimization and is suitable for low-spec devices or real-time tracking situations, whereas the second model is trained in a large dataset and high-performance computing environment to secure high re-identification accuracy and can provide higher discriminability.

[0122] According to one embodiment of the present specification, the processor (28) may calculate a reliability score for each of the first comparison result calculated by comparing the first feature information and the first query feature information, and the second comparison result calculated by comparing the second feature information and the second query feature information. In this case, the comparison result may be adopted only if the reliability score is greater than or equal to a preset threshold. Through this, comparison results with high uncertainty may be excluded in advance.

[0123] Additionally, when the processor (28) adopts both the first comparison result and the second comparison result, it may produce a fused final result by assigning a predefined weight to each comparison result. For example, if the first model is trained for speed optimization and the second model is trained for accuracy improvement, accuracy can be enhanced by assigning a higher weight to the second comparison result. Conversely, in situations where real-time performance is required, the weight of the first comparison result may be assigned more significantly. Therefore, the present specification does not unconditionally use all comparison results of multiple models, but rather selectively uses them according to conditions or fuses them by applying weights, thereby enabling highly reliable object re-identification.

[0124] FIG. 6 is an exemplary diagram illustrating the comparison process between a text-based query and image feature information during model update according to one embodiment of the present specification.

[0125] The image management device (20) can receive and store a first feature vector (61) extracted using the first model (M1) from an image captured by a camera device equipped with the first model (M1, 61). The first model (M1) equipped in the camera device can be updated to a second model (M2, 62). The camera device extracts a second feature vector (62) through the second model (M2) for the captured image and transmits the extracted second feature vector (62) to the image management device (20).

[0126] The image management device (20) can receive a text query (63) from a user. The image management device (20) may further include a text encoder for extracting query feature vectors (64, 65) from the input text query (63).

[0127] The text encoder may include a first text encoder and a second text encoder that generate query feature vectors corresponding to the representation spaces of the first model (M1) and the second model (M2), respectively. The text encoder vectorizes the query sentence. The first text encoder generates a first query feature vector (64) that is aligned with the same representation space as the feature vector generated in the first model (M1), and the second text encoder generates a second query feature vector (65) that is aligned with the same representation space as the feature vector generated in the second model (M2). The first and second query feature vectors (64, 65) can be compared with the first feature vector (61) based on the first model (M1) and the second feature vector based on the second model (M2), respectively, stored in the image management device (20). The image management device (20) can calculate a similarity score from the comparison result and sort the search results (68) based on this score to provide them to the user.

[0128] The text encoder described above can be trained using a dataset of image and text pairs. According to one embodiment, a first model (M1) of a camera device converts an input image into an image feature vector, and a first text encoder converts the input text into a first text feature vector (T1), after which the image feature vector and the first text feature vector can be trained to become close. According to one embodiment, the first text encoder can be implemented to be trained together with the first model (M1) so that a text vector aligned with the feature space of the first model (M1) is generated. The second text encoder can be implemented to be trained together with the second model (M2) so that a text vector aligned with the feature space of the second model (M2) is generated. That is, since the model related to image recognition and the text encoder are trained as a pair, when the image recognition model is newly trained, the text encoder can also be trained together.

[0129] When the model of the camera device is updated from the first model (M1) to the second model (M2), the camera device generates a new feature vector based on the second model (M2), and the first feature vector based on the existing first model (M1) and the second feature vector based on the second model (M2) can coexist in the image management device (20).

[0130] Meanwhile, according to one embodiment, if the second model (M2) is designed to be compatible with the representation space of the first text encoder, even if the first model (M1) is updated to the second model (M2) in the camera device, the first text encoder of the image management device (20) can compare the first feature vector by the first model (M1) with the second feature vector by the second model (M2).

[0131] According to one embodiment, when the first model (M1) in the camera device is updated to the second model (M2), the image management device (20) may additionally be equipped with a second text encoder while the first text encoder remains unchanged.

[0132] Additionally, according to one embodiment, when the first model (M1) is updated to the second model (M2) in the camera device, the second text encoder trained together with the second model (M2) is distributed to the image management device (20) in the form of an update package or provided in the form of a cloud or server Inference API so that it can be accessed by the image management device (20).

[0133] According to one embodiment, when a first model (M1) in a camera device is updated to a second model (M2), when a user inputs a text query (63), the image management device (20) inputs the same query to each of the first text encoder and the second text encoder, compares the first query vector (64) provided by the first text encoder (T1) with the first feature vector (61) by the first model (M1), compares the second query vector (65) provided by the second text encoder (T2) with the second feature vector (62) by the second model (M2), calculates a similarity score from the two comparison results, sorts the search results (68) based on this, and provides them to the user.

[0134] According to one embodiment, when a user inputs a text query such as "A man in black passes through the door of a convenience store," the text query may be used for re-identification purposes (i.e., to re-identify a specific object or person within the image). The input query (63) may be processed by a text encoder. The text encoder may include a first text encoder and a second text encoder, respectively, and may generate a first query feature vector (64) and a second query feature vector (65), respectively, from the same query. The first text encoder generates a vector of the representation space aligned with the first model (M1), and the second text encoder generates a vector of the representation space aligned with the second model (M2).

[0135] FIG. 7 is a flowchart of a method for performing object recoloring according to another embodiment of the present specification.

[0136] FIG. 7 illustrates an embodiment in which the main operator of the object identification process is not a camera device but an image management device (20). In this embodiment, the camera device does not generate object feature information from the captured image and transmits the original image data as is to the image management device (20) (S700). The image management device (20) stores the transmitted original image data and receives an input query for object identification from a user (S710). Subsequently, the image management device (20) determines whether the model used for object identification is the first model (S720).

[0137] When the object identification model is the first model, the image management device (20) applies the first model to the original image data to generate and store first feature information (S731). In addition, the first model is also applied to the input query to generate first query feature information (S733), and the generated first feature information and the first query feature information are compared to determine whether they are the same object (S735).

[0138] On the other hand, when the object identification model is updated from the first model to the second model, the image management device (20) applies the second model to the original image data to generate second feature information (S740). Subsequently, the first model and the second model are applied to the input query, respectively, to generate first and second query feature information (S750), and object identity comparison is performed using these (S760).

[0139] As such, the embodiment of FIG. 7 follows a structure in which feature information generation, query processing, model update response, and feature information comparison are all performed in the image management device (20). Accordingly, since the camera device side performs only image shooting and transmission functions, the burden of computation and firmware update on the camera unit can be reduced. In addition, even when updating the model, the existing stored feature information and the feature information generated by the new model can be managed integrally in the image management device (20), so the continuity and consistency of object identification are maintained even when the model versions are different.

[0140] That is, when the object identification model is updated from the first model to the second model, the image management device (20) can maintain the first feature information without deleting or regenerating it, and can utilize the first feature information together with the second feature information to determine whether they are the same object. The image management device (20) can maintain the existing database containing the first feature information as is without reprocessing. Accordingly, even if the model is updated, the entire existing database does not need to be re-featured, thus providing the effect of reducing computational costs and time.

[0141] One or more of the surveillance cameras, autonomous vehicles, user terminals, and servers may be linked with artificial intelligence modules, robots, augmented reality (AR) devices, virtual reality (VT) devices, devices related to 5G services, etc.

[0142] For the purpose of understanding the present invention, reference numerals have been used in the preferred embodiments illustrated in the drawings, and specific terms have been used to describe the embodiments; however, the present invention is not limited by these specific terms, and the present invention may include all components that are conventionally conceivable by those skilled in the art.

[0143] The present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the present invention may employ direct circuit configurations such as memory, processing, logic, look-up tables, etc., which can execute various functions by the control of one or more microprocessors or other control devices. Similar to how the components of the present invention may be implemented as software programming or software elements, the present invention may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, etc., including various algorithms implemented as combinations of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors. Additionally, the present invention may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations. The above terms may include the meaning of a series of software processes (routines) in conjunction with processors, etc.

[0144] The specific embodiments described in this invention are examples and do not limit the scope of the invention in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as "essential," "importantly," etc., a component may not be strictly necessary for the application of the invention.

[0145] In the specification of the present invention (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in the present invention, it is implied to include an invention applying individual values ​​belonging to said range (unless otherwise stated), and is equivalent to describing each individual value constituting said range in the detailed description of the invention. Finally, regarding the steps constituting the method according to the present invention, unless explicitly stated or otherwise stated, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described.

Claims

1. A storage unit that receives and stores first feature information of an object generated by an artificial intelligence model from a camera device; and A processor that determines whether an input query is the same object by comparing it with first feature information stored in the storage unit, wherein, when the artificial intelligence model mounted on the camera device is updated from the first model to the second model, additionally receives second feature information generated from the updated model and stores it in the storage unit, and applies both the first model and the second model to the query to generate first and second query feature information corresponding to each model, and determines whether the same object is the same by comparing it with the first and second feature information corresponding to each model stored in the storage unit; An image management device including 2. In Paragraph 1, The above processor is, An image management device characterized by providing a re-identification result of the object based on a similarity score of a first comparison result obtained by comparing the first feature information and the first query feature information, and a second comparison result obtained by comparing the second feature information and the second query feature information.

3. In Paragraph 1, The above camera device includes a first model and a second model, and The above processor is, An image management device characterized by receiving the first feature information and the second feature information to which each model is applied, and storing them in the storage unit.

4. In Paragraph 1, The above camera device is, It includes a plurality of camera devices comprising either the first model or the second model, and The above processor is, An image management device characterized by receiving respective feature information generated by the first and second camera devices and storing it in the storage unit.

5. In Paragraph 1, The above query includes video, audio, or text entered by the user, and If the above query is voice or text, An image management device characterized by the above processor generating query feature information from the above query using a generative artificial intelligence model.

6. In Paragraph 5, Management device When the artificial intelligence model mounted on the camera device is updated from the first model to the second model, An image management device characterized by generating first query feature information and second query feature information by a first text encoder learned in conjunction with the first model and a second text encoder learned in conjunction with the second model for the above query.

7. In Paragraph 1, An image management device characterized by the above-mentioned first model and second model generating feature information of different dimensions.

8. In Paragraph 1, An image management device characterized in that the first model is a model trained for speed optimization, and the second model is a model trained for re-identification accuracy improvement.

9. In Paragraph 1, The above processor is, An image management device characterized by receiving only the first feature information extracted from the image data, excluding image data captured by the camera device.

10. In Paragraph 2, The above processor is, An image management device characterized by selecting only the comparison result among the first comparison result and the second comparison result whose reliability score is greater than or equal to a predetermined threshold, or fusing the comparison results according to a predefined weight to determine whether they are the same object.

11. Regarding the method of performing object re-identification, A step of receiving and storing first feature information of an object generated by an artificial intelligence model from a camera device; A step of determining whether an input query is the same object by comparing it with the stored first feature information; When the above artificial intelligence model is updated from a first model to a second model, a step of additionally receiving and storing second feature information generated from the updated model; and A step of applying both the first model and the second model to the above query to generate first and second query feature information corresponding to each model, and determining whether they are the same object by comparing them with the stored first and second feature information, respectively; A method including 12. In Paragraph 11, The step of determining whether the above-mentioned object is the same is, A method characterized by including the step of providing a re-identification result of an object based on a similarity score of a first comparison result obtained by comparing the first feature information and the first query feature information, and a second comparison result obtained by comparing the second feature information and the second query feature information.

13. In Paragraph 11, The above camera device includes a first model and a second model, and A step of receiving and storing first feature information and second feature information to which each model is applied; A method characterized by further including 14. In Paragraph 13, The above camera device includes a plurality of camera devices comprising either the first model or the second model, and A step of receiving and storing respective feature information generated by the first and second camera devices; A method characterized by further including 15. In Paragraph 11, If the above query is video, audio, or text entered by the user, A step of generating query feature information by a generative artificial intelligence model using the above query; A method characterized by further including 16. In Paragraph 11, A method characterized in that the first model and the second model are trained to be different from each other in one or more of the dimension of the feature vector, the type of feature extracted, the network structure, or the learning objective.

17. Regarding the method of performing object re-identification, A step of receiving an image captured from a camera device; A step of generating and storing first feature information by applying a first model to the received image; and The method includes a step of determining whether an input query is the same object by comparing it with the stored first feature information; If the object identification model is updated from the first model to the second model, A step of generating and storing second feature information by applying a second model to the received image; A method further comprising the step of applying both the first model and the second model to the above query to generate first and second query feature information corresponding to each model, and determining whether they are the same object by comparing them with the stored first and second feature information, respectively.

18. In Paragraph 17, If the above object identification model is updated from the above first model to the above second model, A method characterized in that the first feature information is maintained without being deleted or regenerated, and the first feature information is used together with the second feature information to determine whether the object is the same.

Images