Method and system for analyzing image having plurality of objects

Abstract

One embodiment provides a method comprising the steps in which: an image to be analyzed is received and stored in at least one memory; the image to be analyzed is loaded from the at least one memory; at least one processor generates the result of the analysis of the image being analyzed using at least one artificial intelligence model having the image being analyzed as an input, the at least one artificial intelligence model being pre-trained to infer the geometric relationships between a plurality of objects in an image; and the result of the analysis is ingested into at least one post-processing component.

Description

Method and system for analyzing an image containing multiple objects

[0001] The present disclosure relates to a method and system for analyzing an image containing a plurality of objects, and more specifically, to a method and system for analyzing an image containing a plurality of objects that improves the efficiency and accuracy of image analysis by inferring the interaction and spatial relationships between the plurality of objects.

[0002] The field of computer vision has dramatically improved the accuracy of image recognition and analysis alongside the advancement of deep learning technology. In particular, with the introduction of Transformer-based Vision Transformer (ViT) models beyond existing Convolutional Neural Network (CNN)-based models, the ability to model the global context and complex relationships of objects within an image has been greatly enhanced.

[0003] These advancements are increasing the demand for accurate and efficient multi-object analysis systems in various industrial fields requiring advanced object recognition and judgment, such as autonomous driving, medical image analysis, and smart city surveillance.

[0004] Existing image analysis models have exposed significant limitations in inferring complex geometric relationships between multiple objects within an image, namely occlusion or depth order. Conventional approaches have primarily adopted pairwise comparison methods. This requires a quadratic computational cost of n(n-1) / 2 operations to identify all relationships when there are n objects in an image.

[0005] This pairwise approach has caused severe inefficiencies in complex scenes with a large number of objects, to the point where real-time analysis is impossible. Therefore, there is an urgent need to develop a holistic prediction methodology that maintains a constant inference cost regardless of the number of objects in an image.

[0006] Meanwhile, a fundamental inefficiency problem also arises in the tokenization stage, which is the core of the ViT model. ViT simply divides images into patches of a fixed size and converts these patches into tokens. However, this patch-based tokenization often generates a large number of meaningless tokens lacking semantic coherence. For example, simple color blocks in background regions are divided into multiple inefficient tokens. This causes redundant calculations for unnecessary tokens during subsequent attention operations, becoming a major cause of hindering the model's inference speed and overall efficiency.

[0007] To address the inefficiency of tokenization, various model efficiency techniques such as token pruning and token merging have been studied. However, these existing techniques also suffer from problems where semantic criteria are ambiguous, leading to the premature discarding of high-importance tokens or the combination of meaningful features with non-semantic noise, which ultimately undermines the accuracy of image analysis. Therefore, a new method is required to improve efficiency by precisely measuring the semantic importance of tokens and intelligently reconstructing and compressing non-semantic tokens based on these criteria.

[0008] In conclusion, current image analysis technologies fail to efficiently infer object relationships in complex scenes and harbor fundamental problems in which the token sequences underlying the models are constructed inefficiently. In other words, they fail to simultaneously achieve the computational efficiency and semantic refinement suitable for modern AI applications (e.g., autonomous driving, robotics) that require high accuracy and real-time processing capabilities. Consequently, research into new methodologies to overcome these technical shortcomings is necessary.

[0009] According to various embodiments of the present disclosure, the present invention aims to provide a method and system for analyzing an image containing multiple objects that solves the problem of quadratic computational cost occurring in the pairwise inference method of the existing technology and can efficiently identify the geometric relationships (depth and occlusion order) between multiple objects within the image through a single forward pass regardless of the number of objects.

[0010] According to various embodiments of the present disclosure, the present invention aims to provide a method and system for analyzing an image including a plurality of objects that overcomes the problem of semantic inconsistency occurring in patch-based tokenization of an image, maximizes the efficiency of the input token sequence for an image analysis model by precisely evaluating the semantic importance of tokens and intelligently merging and discarding meaningless tokens, and simultaneously improves the inference speed and accuracy of the image analysis model.

[0011] However, the technical problems to be solved by the various embodiments of the present disclosure are not limited to the technical problems described above, and other technical problems may exist.

[0012] One embodiment is,

[0013] A method performed on a computing device comprises the steps of: receiving an image for analysis and storing it in at least one memory; loading the image for analysis from the at least one memory; generating an analysis result for the image for analysis using at least one artificial intelligence model that takes the image for analysis as input by at least one processor, wherein the at least one artificial intelligence model is pre-trained to infer geometric relationships between a plurality of objects within the image; and inputting the analysis result to at least one subsequent processing component.

[0014] In another aspect, the above at least one artificial intelligence model can be pre-trained to infer the geometric relationship through a single inference process.

[0015] In another aspect, the analysis result may include at least one of information regarding the geometric order between multiple objects within the image for analysis and information regarding the shielding relationship between the multiple objects.

[0016] In another aspect, the method may further include the step of the at least one subsequent processing component manifesting the analysis result through at least one user interface.

[0017] In another aspect, the step of expressing the analysis results through at least one user interface may include the step of providing, through the user interface, an image in which different visual effects are applied to each of the plurality of objects in the analysis image according to depth order, based on information regarding the geometric relationships between the plurality of objects in the image.

[0018] In another aspect, the image for analysis may include at least one of an RGB image, an infrared image, a radar image, a single camera image, a stereo camera image, a CT image, an MRI image, an ultrasound image, a satellite image, and a CCTV image.

[0019] In another aspect, the at least one artificial intelligence model may be pre-trained to take features of the image for analysis as input and output a geometric matrix including information regarding the depth of a plurality of objects within the image for analysis or information regarding the occlusion relationship between the plurality of objects.

[0020] In another aspect, the at least one artificial intelligence model is pre-trained by a learning method by the at least one processor, and the learning method comprises: generating a plurality of pixel-wise feature maps from a training image; generating a plurality of mask embeddings by applying a predetermined correlation modeling operation to the plurality of pixel-wise feature maps and a plurality of trainable sub-embeddings; generating a plurality of binary masks through a predetermined operation on the plurality of pixel-wise feature maps and the plurality of mask embeddings; generating a plurality of object region-wise feature maps by applying the plurality of binary masks to the plurality of pixel-wise feature maps; generating a plurality of object-wise global descriptor embeddings using a predetermined operation that models the interrelationships of the plurality of object region-wise feature embeddings; combining a portion of the plurality of mask embeddings corresponding to a plurality of correct objects with the plurality of object-wise global descriptor embeddings; a predetermined operation that models the interaction between the plurality of embeddings included in the combination; dividing the plurality of embeddings generated as a result of the modeling into two groups and generating a predetermined geometric matrix based on the matrix multiplication of the two groups; and based on the geometric matrix and correct data It may include a step of updating the parameters of at least one artificial intelligence model so as to minimize loss.

[0021] In another aspect, the step of generating the plurality of binary masks may include the step of calculating a plurality of mask scores by performing a matrix multiplication operation on the plurality of pixel-specific feature maps and the plurality of mask embeddings, the step of generating a plurality of candidate binary masks through non-linear operations and binarization on the plurality of mask scores, and the step of selecting the plurality of binary masks corresponding to a plurality of correct answer objects by applying an optimal assignment algorithm to the plurality of candidate binary masks.

[0022] In another aspect, the step of generating the plurality of object region-specific feature maps may include the step of generating the plurality of object region-specific feature maps by performing an element-wise multiplication operation on the plurality of binary masks and the plurality of pixel-specific feature maps.

[0023] In another aspect, the step of generating the plurality of object-specific global descriptor embeddings may include the step of performing a predetermined operation to model the interrelationships of the plurality of object-specific feature embeddings, and the step of generating the plurality of object-specific global descriptor embeddings by performing a predetermined pooling operation to compress the dimension of the modeling result.

[0024] In another aspect, some of the plurality of mask embeddings corresponding to a plurality of correct answer objects may be selected through a step of selecting some of the plurality of correct answer objects corresponding to the plurality of correct answer objects by applying an optimal allocation algorithm to the plurality of mask embeddings.

[0025] In another aspect, the step of generating the geometric matrix may include: dividing a plurality of embeddings generated as a result of the modeling of the combination into a first group corresponding to some of the plurality of mask embeddings and a second group corresponding to the plurality of object-specific global descriptor embeddings; obtaining a geometric relation feature representation generated by applying a predetermined non-linear transformation and matrix multiplication operation to each of the first group and the second group; and generating the geometric matrix by performing a predetermined non-linear transformation on the geometric relation feature representation.

[0026] In another aspect, the at least one artificial intelligence model may be pre-trained to calculate semantic importance for a plurality of tokens generated from an image, reconstruct the plurality of tokens by merging some tokens with low importance, and infer geometric relationships between the plurality of entities based on the reconstructed plurality of tokens.

[0027] In another aspect, the at least one artificial intelligence model may include the steps of: generating a plurality of tokens based on an input image; classifying the plurality of tokens into a first group with relatively high importance and a second group with relatively low importance regarding the input image; classifying a plurality of non-meaning tokens included in the second group into a plurality of randomly selected groups; performing a similarity merging operation considering local consistency among the randomly selected groups to convert the plurality of non-meaning tokens into one or more reconstructed tokens; re-evaluating the semantic importance regarding the input image for an entire token sequence including a plurality of tokens included in the first group and one or more reconstructed tokens, and rearranging the entire token sequence based on the re-evaluated importance; and learning to infer geometric relationships between the plurality of objects using the remaining plurality of tokens after removing non-meaning tokens with importance lower than a predetermined threshold from the rearranged entire token sequence.

[0028] In another aspect, the step of classifying the plurality of tokens into the first group and the second group may include the step of calculating the importance between each of the remaining tokens and the entire image token containing information regarding the entire input image included in the plurality of tokens, and the step of classifying tokens whose importance is greater than or equal to a predetermined threshold into the first group and tokens whose importance is less than the predetermined threshold into the second group according to the importance with respect to the entire image token.

[0029] One embodiment is,

[0030] A system is provided comprising at least one memory and at least one processor that executes an image analysis method based on geometric information between a plurality of objects in an image using at least one instruction stored in said memory, wherein the at least one instruction comprises the steps of receiving an image for analysis and storing it in at least one memory; loading the image for analysis from said at least one memory; generating an analysis result for said image for analysis using at least one artificial intelligence model that takes the image for analysis as input by at least one processor, wherein the at least one artificial intelligence model is pre-trained to infer geometric relationships between a plurality of objects in the image; and inputting the analysis result to at least one subsequent processing component.

[0031] In another aspect, the system comprises: a plurality of neurons configured in an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; and at least one routing network that controls the data flow between the plurality of neurons; wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to establish a transmission path for the weights, and may further comprise a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network.

[0032] In another aspect, the system comprises: a plurality of neurons organized into an array including at least one register, at least one microprocessor, and at least one input; and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; wherein each of the plurality of neurons may further comprise an Application Specific Integrated Circuit (ASIC) for a predetermined artificial neural network that is connected to at least one other neuron through any one of the plurality of synapse circuits.

[0033] According to various embodiments of the present disclosure, in image analysis using an image analysis model, the quadratic computation cost incurred in conventional pairwise inference methods can be drastically reduced, thereby achieving a constant inference cost regardless of the number of objects.

[0034] According to various embodiments of the present disclosure, complex geometric relationships (depth and occlusion order) between multiple objects can be accurately and holistically identified using an image analysis model of a single forward pass method.

[0035] According to various embodiments of the present disclosure, the problem of semantic inconsistency of tokens occurring in patch-based tokenization of images can be resolved and semantic efficiency maximized, thereby simultaneously improving the accuracy and inference speed of image analysis.

[0036] According to various embodiments of the present disclosure, through efficient token reconstruction and refined feature modeling of a plurality of tokens obtained from a plurality of patches of an image, high stability and generalization performance can be secured in applications requiring real-time processing, such as autonomous driving.

[0037] However, the effects obtainable through the various embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0038] FIG. 1 illustrates an example of a block diagram of a computing system implementing an analysis service for an image including a plurality of objects according to one embodiment.

[0039] FIG. 2 briefly illustrates the structure of a neuromorphic circuit that may be included in a processor according to one embodiment.

[0040] FIG. 3 is a block diagram of a computing device implementing an analysis service for an image including a plurality of objects according to one embodiment.

[0041] FIG. 4 is a block diagram of a computing device implementing an analysis service for an image including a plurality of objects according to another embodiment.

[0042] FIG. 5 is a block diagram of a computing device implementing an analysis service for an image including a plurality of objects according to another embodiment.

[0043] FIG. 6 is for comparing an image analysis method according to one embodiment with a multi-stage pairwise analysis method that individually infers the relationship between existing object pairs.

[0044] FIG. 7 is a detailed block diagram of an object classification module and an object geometric information generation module included in a computing device according to one embodiment.

[0045] FIG. 8 is intended to illustrate the integrated architecture of an image analysis model according to one embodiment.

[0046] FIG. 9 is intended to illustrate the final analysis results generated for various images according to an image analysis method according to one embodiment.

[0047] FIG. 10 is a detailed block diagram of an image token reconstruction module included in a computing device according to one embodiment.

[0048] FIG. 11 is intended to illustrate the conceptual flow of an image token reconstruction module according to one embodiment.

[0049] FIG. 12 is intended to illustrate the architecture of an image token reconstruction model according to one embodiment.

[0050] FIG. 13 is a flowchart of an image analysis method according to one embodiment.

[0051] FIG. 14 is a flowchart illustrating a learning method for an image analysis model according to one embodiment.

[0052] FIG. 15 is a flowchart illustrating a method for training an image analysis model according to another embodiment.

[0053] As various modifications can be made to the various embodiments of the present disclosure, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the various embodiments of the present disclosure, and the methods for achieving them, will become clear by referring to the embodiments described in detail below together with the drawings. However, the various embodiments of the present disclosure are not limited to the embodiments disclosed below but can be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Also, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Additionally, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the various embodiments of the present disclosure are not necessarily limited to those depicted.

[0054] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0055]

[0056] -Analysis system for images containing multiple objects (1000)

[0057] A system (1000) according to one embodiment provides an artificial intelligence-based image analysis platform that receives an image containing a plurality of objects and provides high-level geometric relationships and quantitative analysis results for said objects. The system (1000) can simultaneously solve the problem of quadratic computational cost arising from the inefficient pairwise inference method of the prior art and the problem of token sequence inconsistency of the Vision Transformer (ViT). Through this, the accuracy of object analysis included in the image can be maximized, and inference speed and efficiency at a level capable of real-time processing can be achieved.

[0058] The system (1000) can identify holistic geometric relationships, such as depth and occlusion order among multiple objects, in a single forward pass manner, regardless of the number of objects included in the image. Additionally, the system (1000) can precisely evaluate the semantic importance of input tokens for an image analysis model and intelligently merge and discard meaningless tokens, thereby eliminating unnecessary calculations for subsequent attention operations and maximizing the semantic efficiency of the token sequence.

[0059] Based on the improved efficiency and accuracy of the image analysis model, the system (1000) can be applied to various application fields requiring high stability and real-time analysis capabilities, such as an object recognition and judgment system for autonomous vehicles, a medical image diagnostic assistance system, and a high-speed object tracking system for smart cities.

[0060] FIG. 1 illustrates an example of a block diagram of a computing system (1000) that implements an analysis service for an image including a plurality of objects according to one embodiment.

[0061] Referring to FIG. 1, a computing system (1000) for implementing an analysis service for an image including a plurality of objects according to one embodiment includes a user computing device (110), a server computing system (130), and a training computing system (150), and the devices can communicate through a network (170).

[0062] A method for analyzing an image including a plurality of objects according to one embodiment may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or implemented and provided by the user computing device (110) and the server computing system (130) in conjunction with each other.

[0063] In this embodiment, the user computing device (110) and / or the server computing system (130) can train a machine learning model (120 and / or 140) through interaction with a training computing system (150) that is communicatedly connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0064] And at this time, the artificial intelligence model can be 1) trained directly locally by a user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training and learning techniques. It may also be implemented by transmitting the artificial intelligence model trained by the training computing system (150) to the user computing device (110) and / or the server computing system (130) through the network (170) to provide / update it.

[0065] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110).

[0066]

[0067] - User Computing Device (110: User Computing Device)

[0068] The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a desktop, a wearable device, an embedded computing device, a tablet PC, an augmented reality (VR) device, and / or a virtual reality (AR) device.

[0069] The user computing device (110) may include at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0070] In particular, according to the embodiment, this processor (111) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which is a hardware technology for implementing a certain digital circuit.

[0071] Here, a field programmable gate array (FPGA) can refer to a flexible digital circuit that is programmable according to user needs.

[0072] As an example, a field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results or state information of operations to support synchronized operation of the FPGA, programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits configurable according to user needs, and an input interface that receives signals from external devices or sensors and transmits them to internal circuits as a channel for receiving data from outside the FPGA.

[0073] Through the combination of the above components, a field-programmable gate array implementation can provide flexible and various types of digital circuits.

[0074] Meanwhile, an Application-Specific Integrated Circuit (ASIC) can refer to a custom integrated circuit that is fixedly designed to perform a specific use or function.

[0075] As an example, the application-dedicated integrated circuit may include a register, which is a small memory device for temporarily storing and managing data and supports the rapid processing of ASIC operations by storing intermediate calculation results or state information; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and coordinates the operation of the entire system by performing various operations or generating control signals when necessary; and an input block, which is an interface for receiving data from the outside, which receives data to be processed by the ASIC and transmits it internally, and receives various input data through connections with sensors or external devices.

[0076] Through the combination of the components mentioned above, an application-specific integrated circuit can perform specific purpose tasks in an optimized manner.

[0077] For example, ASICs can have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits.

[0078] FIG. 2 briefly illustrates the structure of a neuromorphic circuit (300) that may be included in a processor (111, 131, 151) according to one embodiment.

[0079] Referring to FIG. 2, for example, a neuromorphic circuit (300) may include a plurality of presynaptic neuron circuits (310), a plurality of presynaptic lines (311) extending laterally from the plurality of presynaptic neuron circuits (310), a plurality of postsynaptic neuron circuits (320), a plurality of postsynaptic lines (321) extending longitudinally from the plurality of postsynaptic neuron circuits (320), and a plurality of synaptic circuits (330) provided at the intersection of the plurality of presynaptic lines (311) and the plurality of postsynaptic lines (321).

[0080] A plurality of free synaptic neuron circuits (310) can transmit signals input from the outside in the form of electrical signals to a plurality of synaptic circuits (330) through a plurality of free synaptic lines (311).

[0081] Additionally, a plurality of post-synaptic neuron circuits (320) can receive electrical signals from a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0082] Furthermore, multiple post-synaptic neuron circuits (320) may transmit electrical signals to multiple synaptic circuits (330) through multiple post-synaptic lines (321).

[0083] A plurality of synapse circuits (330) can store weights included in layers constituting a neural network system implemented by a neuromorphic circuit (300) and perform a predetermined operation based on the weights and input data.

[0084] For example, each of the plurality of synaptic circuits (330) may include a resistive memory cell having a variable resistance. In this case, the resistance value of the plurality of synaptic circuits (330) changes by a voltage applied through the plurality of presynaptic neuron circuits (310) or the plurality of postsynaptic neuron circuits (320), and can store weight data according to this resistance change.

[0085] The neuromorphic circuit (300) is formed by mimicking the structure of neurons and synapses, which are essential elements of the human brain. When a deep neural network (DNN) is realized using the neuromorphic circuit (300), the data processing speed can be improved and power consumption can be reduced compared to when the existing von Neumann structure is utilized.

[0086] The memory (112) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data (113) and instructions (114) necessary for the at least one processor (111) to perform functional operations such as training an artificial intelligence model or performing high-resolution image analysis through an artificial intelligence model.

[0087] In one embodiment, the user computing device (110) can store at least one machine learning model (120).

[0088] For example, the machine learning model (120) may be various machine learning models, such as multiple neural networks (e.g., deep neural networks) for performing an analysis method on an image containing multiple objects, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0089] For example, machine learning models may include linear regression, decision trees, random forests, gradient-boosting pre-trained language models or / and deep learning models. And neural networks may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or / and other forms of neural networks.

[0090] Additionally, according to various embodiments, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process for an analysis method for an image including a plurality of objects through a large language model (LLM).

[0091] In one embodiment, a user computing device (110) may receive at least one machine learning model (120) from a server computing system (130) via a network (170), store it in memory (112), and then execute the stored machine learning model (120) through a processor (111) to perform an operation for analyzing an image containing a plurality of objects.

[0092] In another embodiment, the server computing system (130) includes at least one machine learning model (140) and performs operations through the machine learning model (140), and can provide an analysis service for an image containing multiple objects to the user by communicating with the user computing device (110) and related data.

[0093] For example, a user computing device (110) can perform an analysis service for an image containing multiple objects in such a way that a server computing system (130) provides an output for the user's input using a machine learning model (140) via the web.

[0094] Additionally, the artificial intelligence model can be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0095] Additionally, the user computing device (110) may include at least one input component (121) for detecting user input. For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) for detecting a touch of a user input medium (e.g., a finger or a stylus), an image sensor for detecting user motion input, a microphone for detecting user voice input, a button, a mouse and / or a keyboard, etc. Additionally, the user input component (121) may include an interface and an external controller when receiving input to an external controller (e.g., a mouse and / or a keyboard, etc.) through an interface.

[0096]

[0097] -Server Computing System (130: Server Computing System)

[0098] The server computing system (130) can perform a series of processes to provide an analysis service for an image containing multiple objects.

[0099] In detail, in an embodiment, the server computing system (130) can provide an analysis service for an image containing multiple objects by exchanging data necessary to enable an analysis service process for an image containing multiple objects to be driven on an external device such as a user computing device (110).

[0100] More specifically, in an embodiment, the server computing system (130) can provide an environment in which an application for providing an analysis service for an image including a plurality of objects on a user computing device (110) can operate.

[0101] To this end, the server computing system (130) may include an application program, data and / or instructions for the application to operate, and may transmit and receive various data based thereon with the external device.

[0102] A server computing system (130) may include at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or electrical units for performing other functions, or a plurality of electrically connected processors.

[0103] For example, ASICs may have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits (see Fig. 2).

[0104] And the memory (132) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations, such as training an artificial intelligence model or executing an analysis method for an image containing multiple objects through the artificial intelligence model.

[0105] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate a plurality of computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include a plurality of computing devices connected to a network (170).

[0106] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer non-linear model as the machine learning model (140). Exemplary neural networks may include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network.

[0107] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, data store) which is a storage for continuously storing and managing raw data that forms the basis of an analysis service for an image including a plurality of objects.

[0108] Such data stores may include various forms of data storage, ranging from file systems to cloud storage. For example, a data store may include at least one database among a relational database that uses a structured query language (SQL) to define and manipulate data, a NoSQL database designed for flexibility and scalability to process unstructured and semi-structured data, a data warehouse optimized for querying and analysis by centralizing large volumes of data from multiple sources as a system used for reporting and data analysis, a data warehouse that stores large volumes of raw data in basic formats such as structured data, semi-structured data, and unstructured data, and a local storage device or Network Attached Storage (NAS) that stores data in files in a format generally accessible by a computer operating system.

[0109]

[0110] - Training Computing System (150: Training Computing System)

[0111] The training computing system (150) may include at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0112] For example, ASICs may have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits (see Fig. 2).

[0113] And the memory (152) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0114] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as back propagation of error.

[0115] For example, such a model trainer (160) can perform updates to one or more parameters of a machine learning model (120 and / or 140) for an analysis service for an image containing multiple objects based on a defined loss function in a backpropagation manner.

[0116] In some embodiments, performing backpropagation of the error may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight devaluation, dropout and / or knowledge distillation, etc.) to improve the generalization ability of the machine learning model (120 and / or 140) being trained.

[0117] For example, a model trainer (160) can train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data of different forms, such as, for example, images, audio samples and / or text.

[0118] Examples of image types that can be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0119] These training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0120] And the model trainer (160) includes computer logic that is utilized to provide the desired function.

[0121] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one embodiment, the model trainer (160) may include a program file stored in a storage device, be loaded into memory (152), and be executed by one or more processors (151). In another embodiment, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0122] Network (170) includes, but is not limited to, 3GPP (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth network, satellite broadcasting network, analog broadcasting network and / or DMB (Digital Multimedia Broadcasting) network.

[0123] Generally, communication through the network (170) can be performed using any type of wired and / or wireless connection through various communication protocols (e.g., TCP / IP, HTTP, SMTP and / or FTP, etc.), encodings or formats (e.g., HTML and / or XML, etc.), and / or protection schemes (e.g., VPN, Secure HTTP and / or SSL, etc.).

[0124] FIG. 3 is a block diagram of a computing device (100) that implements an analysis service for an image including a plurality of objects according to one embodiment.

[0125] Referring to FIG. 3, the computing device (100) included in the user computing device (110), server computing system (130), and training computing system (150) includes a plurality of applications (e.g., applications 1 to N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing application (e.g., detection, classification, and / or segmentation, etc.), a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0126] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and by storing and operating the trained artificial intelligence model, it may provide output data according to a predetermined input data.

[0127] Each application of the computing device (100) can communicate with a number of other components of the computing device (100), such as, for example, at least one sensor, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0128] FIG. 4 is a block diagram of a computing device (200) that implements an analysis service for an image including a plurality of objects according to another embodiment.

[0129] Referring to FIG. 4, the computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application can communicate with a central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application and / or a browser application, etc. In one embodiment, each application can communicate with the central intelligence layer (and a model stored therein) using an API (e.g., a common API across all applications).

[0130] The central intelligence layer may include a number of machine learning models. For example, as illustrated in FIG. 4, at least some of the machine learning models may be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model for all applications. In some embodiments, the central intelligence layer may be included within the operating system of the computing device (200) or otherwise implemented.

[0131] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device (200). As illustrated in FIG. 4, the central device data layer can communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0132] The technology described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from said systems. It will be recognized that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, division of tasks, and functionality between and from components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0133] FIG. 5 is a block diagram of a computing device (400) implementing an analysis service for an image containing a plurality of objects according to another embodiment. FIG. 6 is for comparing an image analysis method according to an embodiment with a multi-stage pairwise analysis method that individually infers the relationship between existing object pairs. FIG. 7 is a detailed block diagram of an object classification module (10) and an object geometric information generation module (20) included in a computing device (400) according to an embodiment. FIG. 8 is for explaining the integrated architecture of an image analysis model according to an embodiment. FIG. 9 is for explaining the final analysis results generated for various images according to an image analysis method according to an embodiment. FIG. 10 is a detailed block diagram of an image token reconstruction module (40) included in a computing device (400) according to an embodiment. FIG. 11 is for explaining the conceptual flow of an image token reconstruction module according to an embodiment. FIG. 12 is for explaining the architecture of an image token reconstruction model according to an embodiment.

[0134] Referring to FIG. 5, the computing device (400) included in the user computing device (110), server computing system (130) and training computing system (150) may include an object classification module (10), an object geometric information generation module (20), an analysis result provision module (30), and an image token reconstruction module (40).

[0135] The object classification module (10) can perform the function of extracting features from an input image and generating object candidate information. Through this, the object classification module (10) can generate basic feature information necessary for geometric relationship inference for multiple objects included in the input image.

[0136] The object geometric information generation module (20) can infer complex spatial relationships between multiple objects based on feature information generated by the object classification module (10). The object geometric information generation module (20) can produce a geometric matrix regarding at least one of depth relationships and occlusion order holistically for an input image regardless of the number of objects, using a single forward pass method, moving away from the conventional pairwise inference method.

[0137] For example, referring to Fig. 6, conventional pairwise inference methods (Pairwise Nets, VLM, Foundation Models, Mask2Order) determine the relationship between multiple objects (A, B) individually and require a multi-stage inference process proportional to the number of object pairs. This causes inefficiency, where computational costs increase quadratically as the number of objects increases.

[0138] On the other hand, the object geometric information generation module (20) according to one embodiment adopts a holistic inference method, unlike the existing method. The object geometric information generation module (20) can produce a geometric matrix based on at least one of depth relationship and occlusion order holistically for an input image regardless of the number of objects through a Single Forward Pass method, moving away from the existing pairwise inference method. This solves the problem of computational cost associated with multi-stage pairwise analysis methods that infer relationships between object pairs individually, and can dramatically increase the possibility of real-time analysis.

[0139] The object classification module (10) and the object geometric information generation module (20) cooperate closely through a clear division of roles involving data supply and analysis. When the object classification module (10) generates refined feature information representing the identity and location of an object from an input image, the object geometric information generation module (20) can receive this feature information and perform a final analysis task of modeling the relationship between pairs of objects and generating a geometric matrix. In this way, the object classification module (10) serves as an indispensable preprocessing and feature source for the efficient inference of geometric relationships between multiple objects by the object geometric information generation module (20).

[0140] For example, referring to FIG. 7, the object classification module (10) may include an image feature map generation module (11) and a mask embedding generation module (12).

[0141] The image feature map generation module (11) can generate multiple pixel-by-pixel feature maps from the input image.

[0142] For example, referring to FIG. 8, the image feature map generation module (11) may include a backbone that extracts hierarchical features of an input image and a per-pixel decoder that integrates the output of the backbone to restore high-resolution features. The backbone expands three-channel information of the input image into multiple channels (C), and the per-pixel decoder integrates the multiple-layer features of the backbone to output a per-pixel feature map (P) containing spatial location information of the image.

[0143] The mask embedding generation module (12) can generate a mask embedding containing the identity and shape features of an object based on the features of an input image.

[0144] For example, referring to FIG. 8, the mask embedding generation module (12) may include a plurality of transformer blocks that perform iterative cross attention operations on a plurality of learnable sub-embeddings (L-sub) and a plurality of pixel-wise feature maps (P) from the image feature map generation module (11). In this case, a plurality of learnable sub-embeddings (L-sub) may be used as a query, and a plurality of pixel-wise feature maps (P) may be used as a key and a value. Through this process, a mask embedding (Q) condensed with the identity and shape features of an object included in the input image may be generated. This mask embedding (Q) is provided as a core input to the subsequent modules, the binary mask generation module (21) and the geometric matrix generation module (24).

[0145] Additionally, for example, referring to FIG. 7, the object geometric information generation module (20) may include a binary mask generation module (21), an object region-specific feature map generation module (22), an object-specific global descriptor generation module (23), and a geometric matrix generation module (24).

[0146] The binary mask generation module (21) can generate multiple binary masks (M') by receiving multiple pixel-specific feature maps (P) and multiple mask embeddings (Q) generated by the object classification module (10).

[0147] For example, referring to FIG. 8, the binary mask generation module (21) can generate multiple mask scores by performing matrix multiplication operations on multiple pixel-specific feature maps (P) and multiple mask embeddings (Q), and generate N candidate binary masks (M) through non-linear operations and binarization on the multiple mask scores.

[0148] Furthermore, the binary mask generation module (21) can apply an Optimal Assignment Algorithm to the generated N candidate binary masks (M) to select only n valid masks corresponding to multiple correct answer objects, and provide the selected n valid masks as the final binary masks (M') to be used by the subsequent module for analysis.

[0149] Here, the optimal allocation algorithm may include a Hungarian Matcher method that performs an operation to find the best one-to-one correspondence between predicted segments and ground-truth data with minimum cost.

[0150] For example, the optimal allocation algorithm can generate a cost matrix for pairing three prediction segments (M1, M2, M3) and two ground truth data (G1, G2). Each element (Cij) of the cost matrix can be calculated by weighted summing the locality mismatch cost (IoU Loss) and the class classification error cost between the prediction segment (Mi) and the ground truth data (Gj).

[0151] The Hungarian matcher finds the optimal one-to-one correspondence that minimizes the sum of the total costs based on this cost matrix. If the predicted segments M2 and M3 are assigned to the ground truth data G1 and G2, respectively, at the lowest cost, the algorithm confirms the pairs (M2, G1) and (M3, G2) as final valid matches. At this time, the predicted segment M1, which is not assigned and remains, is classified as 'No Object' and discarded, so the model can retain only two valid segments necessary for subsequent analysis.

[0152] The object region-specific feature map generation module (22) can separate only the valid features corresponding to the object region among the features contained in the multiple pixel-specific feature maps (P) by using the input multiple binary masks (M') and multiple pixel-specific feature maps (P).

[0153] For example, referring to FIG. 8, the object region feature map generation module (22) can generate multiple object region feature maps (P') by performing element-wise multiplication operations on multiple binary masks (M') and multiple pixel feature maps (P). Through element-wise multiplication operations, pixel feature maps (P) corresponding to the 0 value (background) of the mask are removed, and only valid features corresponding to the 1 value (object region) of the mask are left, thereby providing refined object region features with background noise removed for analysis by a subsequent module.

[0154] The object-specific global descriptor generation module (23) can generate a global descriptor, which is a single vector condensing the spatial features of each object, based on a plurality of object-specific feature maps (P') generated by the object-specific feature map generation module (22).

[0155] For example, referring to FIG. 8, the object-specific global descriptor generation module (23) can perform a predetermined operation to model the interrelationships of multiple object-specific feature maps (P'). For example, the predetermined operation may include a self-attention operation using a Transformer Encoder, and through this operation, pixel information within the object area is integrated and refined.

[0156] In this case, a masked self-attention operation according to the following equations (1) and (2) can be applied to multiple object region feature maps (P'). Accordingly, background features outside the object region are excluded from the operation, and a global descriptor can be generated by integrating only the features inside the object without the influence of background noise on the multiple object region feature maps (P').

[0157]

[0158] Equation (1):

[0159] Equation (2): , / , otherwise.

[0160]

[0161] Furthermore, the object-specific global descriptor generation module (23) can perform a predetermined pooling operation (2D-Pooling) to compress the dimensions of the modeling result, thereby finally generating a plurality of object-specific global descriptor embeddings (D) with spatial dimensions removed. The plurality of object-specific global descriptor embeddings (D) can be combined with a mask embedding (Q) in a subsequent module, the geometric matrix generation module (24), and used for geometric relationship inference.

[0162] The geometric matrix generation module (24) can generate a geometric matrix containing final geometric relationship information between multiple objects included in the input image by integrating object-specific features and modeling interactions.

[0163] The geometric matrix generation module (24) can combine a selected portion of the multiple mask embeddings (Q) provided by the object classification module (10) and the object-specific global descriptor generation module (23) with multiple object-specific global descriptor embeddings (D).

[0164] In this case, for example, the geometric matrix generation module (24) can concatenate a plurality of mask embeddings (Q') selected through an Optimal Assignment Algorithm among a plurality of mask embeddings (Q) with a plurality of object-specific global descriptor embeddings (D).

[0165] Here, multiple object-specific global descriptor embeddings (D) contain spatially-grounded information, and selected multiple mask embeddings (Q') contain geometric and shape-relevant knowledge, so by combining them, all key clues necessary for inferring geometric relationships between objects can be obtained.

[0166] For example, referring to FIG. 8, the geometric matrix generation module (24) can perform a predetermined operation to model the interaction between a plurality of embeddings included in the combination.

[0167] For example, the above-mentioned predetermined operation may include a self-attention operation using a Transformer Decoder.

[0168] Subsequently, the geometric matrix generation module (24) divides the plurality of embeddings generated from the modeling result into two groups (Q L , D L ) can be divided into. For example, the geometric matrix generation module (24) can divide the plurality of embeddings generated as a modeling result for the combination into a first group (Q') corresponding to the selected plurality of mask embeddings (Q'). L) and a second group (D) corresponding to multiple object-specific global descriptor embeddings (D). L It can be divided into ).

[0169] In addition, the geometric matrix generation module (24) consists of two groups (Q L , D L After applying a non-linear transformation to each, a matrix multiplication operation can be performed to obtain a geometric relation feature representation.

[0170] Furthermore, the geometric matrix generation module (24) can generate a geometric matrix by performing a predetermined non-linear transformation on the geometric relationship feature representation. In this case, the geometric matrix generation module (24) can generate a first geometric matrix (Gd) containing depth relationship information and a second geometric matrix (Go) containing occlusion relationship information by applying a multilayer perceptron with different parameters to the geometric relationship feature representation.

[0171] Afterward, at least one processor of the computing device (400) can update the parameters of at least one artificial intelligence model that has performed the series of operations to minimize the two loss functions (Ld, Lo) for the correct data of the two geometric matrices, thereby enabling pre-training of the model.

[0172] The analysis result providing module (30) can perform inference to analyze geometric relationships between multiple objects for an input image for analysis using a pre-trained image analysis model as described above.

[0173] Referring to FIG. 9, the analysis result providing module (30) can perform the function of providing the final analysis result to a user interface or a subsequent processing component based on the inference result. The analysis result providing module (30) can express the analysis result intuitively and practically by expressing the inferred depth relationship and occlusion relationship information in the form of a visual graph, or by providing an image in which different visual effects (e.g., highlights) are applied according to the depth order to each of the multiple objects in the image.

[0174] For example, the analysis result providing module (30) can clearly express the analysis result by distinguishing between objects located in front and objects located behind based on the inferred depth order and applying visual effects such as brightness or color differentially to each object area.

[0175] Additionally, for example, the image for analysis may include at least one of an RGB image, a single camera image, a stereo camera image, a CT image, an MRI image, an ultrasound image, a satellite image, and a CCTV image.

[0176] Meanwhile, the image token reconstruction module (40) calculates the semantic importance of a plurality of tokens generated from an input image and performs a predetermined reconstruction operation to reconstruct the plurality of tokens by merging some tokens with low importance, thereby selecting some tokens with relatively high importance to use in an operation for image analysis. Through this, the image token reconstruction module (40) can maximize token processing efficiency and improve the speed and accuracy of subsequent analysis operations.

[0177] For example, referring to FIG. 10, the image token reconstruction module (40) may include a token generation module (41), a token group classification module (42), a token merging module (43), and a semantic token determination module (44). The image token reconstruction module (40) can mitigate semantic inconsistency issues that occur in the initial patch-based tokenization of ViTs.

[0178] Referring to FIG. 11, the conceptual flow of re-tokenization performed by the image token reconstruction module (40) can be seen. Initial tokens generated through the tokenizer often contain non-meaningful features at the individual patch level, such as in region (a). The image token reconstruction module (40) can process these tokens to convert them into tokens that have acquired meaning after re-tokenization, such as in (b), or finally discard tokens that are still non-meaningful, such as in (c), thereby drastically reducing the number of unnecessary tokens.

[0179] For example, referring to FIG. 12, the image token reconstruction module (40) can generate multiple tokens corresponding to multiple patches for an input image using the token generation module (41).

[0180] Additionally, the token group classification module (42) can classify multiple tokens into a first group with relatively high importance to the input image and a second group with relatively low importance.

[0181] In this case, for example, the token group classification module (42) can calculate the importance between the entire image token containing information about the entire input image included in the plurality of tokens and each of the remaining tokens. Here, the entire image token refers to a CLS token, and the importance may refer to a CLS attention score.

[0182] The token group classification module (42) can classify tokens whose importance for the entire image token of each of these tokens is greater than or equal to a predetermined threshold into a first group, and tokens whose importance is less than a predetermined threshold into a second group. In this way, the token group classification module (42) can classify multiple tokens into semantic tokens of relatively high importance and non-semantic tokens of relatively low importance based on CLS attention scores.

[0183] Afterwards, the token merging module (43) can arbitrarily classify the selected meaningless tokens into multiple groups. For example, the token merging module (43) can divide the selected meaningless tokens into Group A and Group B.

[0184] Additionally, the token merging module (43) can perform a similarity merging operation (e.g., bipartite matching) that considers local coherence for Group A and Group B, and through this process, non-meaning tokens can be reconstructed while acquiring context.

[0185] For example, the token merging module (43) may consider both feature similarity scores and distance information to merge meaningless tokens (TA) belonging to Group A and meaningless tokens (TB) belonging to Group B. The token merging module (43) first considers the feature similarity (S) between TA and TB. feature Calculate ), and separately, based on the location embeddings of TA and TB, the physical distance between the two tokens (D dis ) calculates. Afterwards, this physical distance (D dis Weight (W) inversely proportional to ) dis ) is calculated and combined with feature similarity to obtain the final enhanced similarity score (S enhanced ) can be done (S enhanced ∝ S feature × W dis ).

[0186] Through this process, a bias is applied so that the similarity score increases as spatial proximity increases (the shorter the distance), and optimal merged pairs can be determined by performing bipartite matching based on this enhanced score.

[0187] Subsequently, the token merging module (43) can re-evaluate the semantic importance of the entire image token (CLS token) of the entire input token sequence, including semantic tokens and merged and reconstructed tokens, and classify them again into a first group and a second group according to the order of importance. The merging and re-evaluation described above can be repeatedly performed on the second group, which is again determined to have low importance.

[0188] The semantic token determination module (44) can perform analysis on the input image using the remaining multiple tokens after repeatedly discarding tokens from the token sequence that have an importance lower than a predetermined threshold through merging and re-evaluation. In this way, the overall inference speed and efficiency of the system can be maximized by passing only the minimum number of valid tokens to subsequent analysis operations.

[0189] For example, the input token sequence in which token reconstruction is completed in this way can be used to infer geometric relationships between objects in an image through an object classification module (10) and an object geometric information generation module (20).

[0190]

[0191] - Analysis method for an image containing multiple objects (S100)

[0192] A method for analyzing an image containing a plurality of objects according to one embodiment (S100) receives an image containing a plurality of objects and can efficiently and accurately infer geometric relationships between the objects. The method (S100) aims to simultaneously overcome the inefficiency of feature extraction and the computational limitations of complex relationship inference inherent in existing Vision Transformer (ViT) models. Through the method (S100), computational costs can be reduced through holistic relationship inference based on a single inference process, and input features based on semantic importance can be optimized.

[0193] According to the method (S100), a single inference process (Single Forward Pass) can be performed regardless of the number of objects in the image to produce a geometric matrix containing depth order and occlusion relationships for all pairs of objects.

[0194] The generated geometric matrix quantitatively represents the spatial relationship between multiple objects, and analysis results can be provided based on this information. These analysis results can be displayed in the form of images with visual effects applied according to depth order through a user interface, and can be immediately input and utilized in various subsequent application systems requiring real-time performance and high accuracy, such as autonomous driving, robotics, and medical image analysis.

[0195] Additionally, according to the method (S100), relatively low-importance, meaningless tokens are selected based on tokens generated from an input image, and the efficiency of the token sequence can be maximized by reconstructing them through similarity merging considering local coherence. This optimized feature sequence plays a role in increasing the inference speed of the model, and at least one artificial intelligence model can produce a geometric matrix including depth order and occlusion relationships for all object pairs within the image based on this optimized feature information.

[0196] FIG. 13 is a flowchart of an image analysis method (S100) according to one embodiment. FIG. 14 is a flowchart for explaining a method for learning an image analysis model according to one embodiment. FIG. 15 is a flowchart for explaining a method for learning an image analysis model according to another embodiment.

[0197] Referring to FIG. 13, an image analysis method (S100) according to one embodiment may include the steps of receiving an image for analysis and storing it in at least one memory (S101), loading the image for analysis from the at least one memory (S103), generating an analysis result for the image for analysis using at least one artificial intelligence model that takes the image for analysis as input by at least one processor (S105), and inputting the analysis result to at least one subsequent processing component (S107).

[0198] In one embodiment, the method (S100) may be performed by a processor (131) included in a server computing system (130). However, it is not limited thereto, and at least a part of the method (S100) may be performed by a processor (111) of a user computing device (110) or a processor (151) of a training computing system (150), and another part may be performed by a processor (131) included in a server computing system (130).

[0199] For convenience of explanation, the following description describes a processor (131) included in a server computing system (130) performing the method (S100).

[0200] In step (S101), the processor (131) may receive an image for analysis via an external device or network and store it in at least one memory so that it can be used for analysis. The image for analysis may be an image containing multiple objects, such as a single camera image of an autonomous vehicle, a CT image of medical equipment, or a CCTV image of a smart city, and corresponds to target data for which an artificial intelligence model infers geometric relationships. The memory may include any form of storage medium accessible to the processor (131), such as volatile memory (RAM) or a non-volatile storage device.

[0201] The processor (131) can convert the received data to the input format of an artificial intelligence model or perform preprocessing so that it can be used in a subsequent step of an image analysis method. For example, it can standardize the format of the image or resize it. Through these initial steps, the system (1000) can prepare the data to be analyzed and secure data accessibility to begin high-speed analysis of multiple objects.

[0202] In step (S103), the processor (131) may load an image for analysis stored in at least one memory for subsequent processing. Step (S103) may include loading data from a storage location into the processor's (131) workspace (e.g., cache or main memory) so that the processor (131) can directly access and process the image for analysis.

[0203] In step (S105), the processor (131) can perform a series of key processes to generate analysis results for an image for analysis using an artificial intelligence model.

[0204] For example, in step (S105), the processor (131) can sequentially drive core modules such as the object classification module (10) and the object geometric information generation module (20) to infer geometric relationships between multiple objects within an image through a single inference process (Single Forward Pass). In this inference process, the processor (131) can generate a geometric matrix containing depth order and occlusion relationship information, and can generate a final analysis result based on the generated geometric matrix. For example, in step (S105), the processor (131) can generate a final analysis result based on the inference result using the analysis result providing module (30).

[0205] Referring to FIG. 14, an artificial intelligence model according to one embodiment comprises the steps of: generating a plurality of pixel-wise feature maps from a training image (S1051); generating a plurality of mask embeddings by applying a predetermined correlation modeling operation to the plurality of pixel-wise feature maps and a plurality of trainable sub-embeddings (S1053); generating a plurality of binary masks through a predetermined operation on the plurality of pixel-wise feature maps and the plurality of mask embeddings (S1055); generating a plurality of object region-wise feature maps by applying the plurality of binary masks to the plurality of pixel-wise feature maps (S1057); generating a plurality of object-wise global descriptor embeddings using a predetermined operation that models the interrelationships of the plurality of object region-wise feature embeddings (S1059); combining a portion of the plurality of mask embeddings corresponding to a plurality of correct objects with the plurality of object-wise global descriptor embeddings (S1061); performing a predetermined operation that models the interaction between the plurality of embeddings included in the combination; dividing the plurality of embeddings generated as a result of the modeling into two groups, and for the two groups It can be pre-trained by a learning method comprising the step of generating a predetermined geometric matrix based on matrix multiplication (S1063), and the step of updating the parameters of at least one artificial intelligence model so as to minimize loss based on the geometric matrix and the correct answer data (S1065).

[0206] In step (S1051), the processor (131) can generate multiple pixel-by-pixel feature maps (P) for an input training image using the backbone of the artificial intelligence model and a per-pixel decoder. Through the above process, the three-channel information of the image is expanded into feature information having multiple channels (C), and a dense feature representation containing spatial position information of the entire image and deepened visual features is produced, thereby preparing basic data required for subsequent steps.

[0207] In step (S1053), the processor (131) can generate a plurality of mask embeddings (Q) using a plurality of pixel-wise feature maps (P) produced by the image feature map generation module (11) and a plurality of learnable sub-embeddings (L-sub). In the above process, the processor (131) can generate a plurality of mask embeddings (Q) in which the identity and shape features of an object are condensed by repeatedly performing cross-attention operations on the plurality of sub-embeddings (L-sub) by referencing the plurality of pixel-wise feature maps (P) as key and value sources.

[0208] In step (S1055), the processor (131) can generate a plurality of binary masks (M') through a predetermined operation on a plurality of pixel-wise feature maps (P) calculated by the image feature map generation module (11) and a plurality of mask embeddings (Q) generated in step (S1053).

[0209] For example, the processor (131) can generate multiple mask scores by performing matrix multiplication operations on multiple mask embeddings (Q) and multiple pixel-specific feature maps (P), and generate N candidate binary masks (M) through non-linear operations and binarization on the multiple mask scores.

[0210] Furthermore, the processor (131) can use the binary mask generation module (21) to apply an Optimal Assignment Algorithm to N candidate binary masks (M) to select only n valid masks corresponding to multiple correct answer objects, and provide the selected n valid masks as the final binary masks (M') to be used by the subsequent module for analysis.

[0211] In step (S1057), the processor (131) can generate multiple object region feature maps (P') by applying multiple binary masks (M) to multiple pixel region feature maps (P) through element-wise multiplication operations. Through element-wise multiplication operations, pixel region feature maps (P) corresponding to the 0 value (background) of the mask are removed, and only valid features corresponding to the 1 value (object region) of the mask are left, thereby providing refined object region features with background noise removed for analysis by a subsequent module.

[0212] In step (S1059), the processor (131) can generate a plurality of object-specific global descriptor embeddings (D) using a plurality of object-specific feature maps (P') generated in step (S1057). The processor (131) can generate a plurality of object-specific global descriptor embeddings (D) by sequentially applying a predetermined operation (Transformer Encoder) that models interrelationships within object regions and a predetermined pooling operation (2D-Pooling) that compresses dimensions to the plurality of object-specific feature maps (P').

[0213] In step (S1061), the processor (131) can combine a plurality of mask embeddings (Q') selected from a plurality of mask embeddings (Q) through an Optimal Assignment Algorithm with a plurality of object-specific global descriptor (D) embeddings. Through this combination operation, a single feature vector is generated that integrates the identity and shape characteristics of the objects with their spatial context, which can provide all the key information necessary to infer relationships between objects in subsequent steps.

[0214] In step (S1063), the processor (131) may perform a predetermined operation by a transformer decoder to model the interaction between a plurality of embeddings included in the combination.

[0215] Subsequently, the processor (131) divides the plurality of embeddings generated from the modeling result into two groups (Q L , D L A geometric relation feature representation can be obtained by dividing into ) and performing matrix multiplication operations on the two groups.

[0216] Furthermore, the processor (131) may generate a geometric matrix by performing a predetermined non-linear transformation on the geometric relation feature representation. In this case, the processor (131) may generate a first geometric matrix (Gd) containing depth relation information and a second geometric matrix (Go) containing occlusion relation information by applying a multilayer perceptron with different parameters to the geometric relation feature representation.

[0217] In step (S1065), the processor (131) can update the parameters of at least one artificial intelligence model that has performed the series of operations to minimize the two loss functions (Ld, Lo) for the correct data of the two geometric matrices, thereby enabling pre-training of the model.

[0218] In step (S107), the processor (131) may provide the analysis results from step (S105) to a user interface or a subsequent processing component. The processor (131) may use the analysis result providing module (30) to represent the inferred depth relationship and occlusion relationship information in the form of a visual graph, or provide an image in which different visual effects (e.g., highlights) are applied to each of the multiple objects in the image according to the depth order, thereby making the analysis results intuitive and usable.

[0219] For example, the image processed by the processor (131) may include medical images such as CT, MRI, or ultrasound images.

[0220] In this case, the processor (131) can use a geometric matrix generated from the image to identify multiple objects in vivo, such as a tumor and surrounding tissues within the image, and quantify the distance, size, and positional relationship between these objects to determine whether the lesion is malignant.

[0221] Additionally, the processor (131) can perform the task of setting the path of the surgical robot by analyzing the relative positions of organs based on the analysis of the 3D image.

[0222] Furthermore, the processor (131) can monitor the progression of the disease in real time by comparing the analysis results of the previous image with the analysis results of the current image to track the movement, deformation, and size change of the organ, and can provide the results in real time through a user interface.

[0223] In addition, for example, image analysis by the processor (131) can be applied to smart factory and industrial automation systems to improve safety management and production efficiency.

[0224] For example, the processor (131) can perform a task of maintaining a safe distance between the worker and the robot and setting a path to avoid obstacles based on an analysis of an image taken inside a factory where the robot and the worker coexist.

[0225] Additionally, the processor (131) can perform a task of detecting signs of failure, such as wear or detachment of equipment, at an early stage by continuously detecting changes in the relative positions between parts of the production equipment based on an analysis of an image of the production equipment.

[0226] Furthermore, the processor (131) can implement an automatic inventory management function by analyzing and monitoring the location, quantity, and placement status of items in real time based on an analysis of an image of items displayed in a warehouse.

[0227] Additionally, for example, image analysis by the processor (131) can improve the accuracy of spatial alignment between the real world and virtual objects, which is essential for creating virtual / augmented reality content.

[0228] For example, the processor (131) can receive an image of a real space captured by a user, infer depth and occlusion order information between multiple real objects, and perform a task of aligning a virtual object to an exactly intended location in the real space based on this.

[0229] Furthermore, the processor (131) can perform the task of analyzing an image of a real space taken by a user and rapidly generating a game map or virtual environment that reflects the spatial characteristics.

[0230] In addition, for example, image analysis by the processor (131) makes it possible to obtain precise spatial information in the field of 3D mapping and autonomous flight using a drone.

[0231] For example, the processor (131) can perform the task of analyzing an image taken by the drone to recognize the 3D location and size of an obstacle (building, tree, etc.) and to generate a safe flight path to avoid it.

[0232] Furthermore, the processor (131) can perform the task of accurately analyzing geometric information between various objects constituting the city based on multiple images captured by the drone, and precisely generating a 3D model of the city based on this.

[0233] Additionally, the processor (131) can perform the task of quickly identifying the location of the rescue target by analyzing images of the disaster site to support rescue activities and generating a sophisticated real-time 3D map including various obstacles at the disaster site.

[0234] In this way, the processor (131) can infer geometric relationships between multiple objects included in an image through analysis of the image, and the result of this inference can be effectively applied to various industrial fields requiring high reliability and real-time processing, such as object judgment in autonomous vehicles, robot path planning in smart factories, medical image analysis, and augmented reality content generation, thereby significantly improving the overall performance and utility of the system.

[0235] Meanwhile, referring to FIG. 15, an artificial intelligence model according to another embodiment may be pre-trained by a learning method comprising the steps of: generating a plurality of tokens based on an input image (S151); classifying the plurality of tokens into a first group with relatively high importance for the input image and a second group with relatively low importance (S153); classifying a plurality of non-meaning tokens included in the second group into a plurality of groups arbitrarily (S155); performing a similarity merging operation considering local consistency among the arbitrarily classified plurality of groups to convert the plurality of non-meaning tokens into one or more reconstructed tokens (S157); re-evaluating the semantic importance for the input image for an entire token sequence including a plurality of tokens included in the first group and one or more reconstructed tokens, and rearranging the entire token sequence based on the re-evaluated importance (S159); and learning to infer geometric relationships between the plurality of objects using the remaining plurality of tokens after removing non-meaning tokens with importance lower than a predetermined threshold from the rearranged entire token sequence (S161).

[0236] In step (S151), the processor (131) may, for example, divide the input image into patches using a tokenizer of the ViT model and convert the multiple patches into multiple tokens to generate a token sequence.

[0237] In step (S153), the semantic importance of each of the input images of the plurality of tokens is evaluated, and the tokens can be classified into a first group (meaningful tokens) with relatively high importance and a second group (non-meaningful tokens) with low importance.

[0238] In step (S155), the processor (131) may classify a plurality of non-meaning tokens included in the second group into a plurality of groups (e.g., Group A and Group B) as a preparatory step for similarity merging.

[0239] In step (S157), the processor (131) can perform a similarity merging operation that considers local coherence among the arbitrarily classified plurality of groups to convert a plurality of meaningless tokens into one or more reconstructed tokens.

[0240] In step (S159), the processor (131) can re-evaluate the semantic importance of the entire token sequence, which includes the first group of tokens and one or more reconfigured tokens, and rearrange the entire token sequence based on the re-evaluated importance to identify the final disposal target.

[0241] Meanwhile, steps (S153) through (S159) may be performed repeatedly until a predetermined condition is satisfied. For example, they may be performed until a preset maximum number of iterations (K) is satisfied. Alternatively, the iteration may be terminated and the process proceed to the next step (S161) when the total number of rearranged token sequences no longer decreases below a predetermined threshold (i.e., when semantic efficiency converges).

[0242] In step (S161), the processor (131) can remove meaningless tokens from the rearranged token sequence that have an importance lower than a predetermined threshold and train a model to infer geometric relationships between multiple objects using the remaining multiple valid tokens.

[0243] Various embodiments of the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the various embodiments of the present disclosure, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the various embodiments of the present disclosure, and vice versa.

[0244] The specific embodiments described in this disclosure are exemplary and do not limit the scope of the various embodiments of this disclosure in any way. For the sake of brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are exemplary representations of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of the various embodiments of this disclosure.

[0245] Furthermore, although the detailed description of the present disclosure has been described with reference to preferred embodiments of the present disclosure, those skilled in the art or those with ordinary knowledge in the art will understand that various modifications and changes can be made to the various embodiments of the present disclosure without departing from the spirit and technical scope of the various embodiments of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the various embodiments of the present disclosure should not be limited to the contents described in the detailed description of the specification but should be determined by the claims.

[0246] Various embodiments of the present disclosure have industrial applicability in that they can achieve a constant inference cost regardless of the number of objects by drastically reducing the quadratic computation costs incurred in conventional pairwise inference methods in image analysis using artificial intelligence models.

Claims

1. A method performed on a computing device, A step of receiving an image for analysis and storing it in at least one memory; A step of loading the analysis image from at least one memory; A step in which at least one processor uses at least one artificial intelligence model that takes the analysis image as input to generate an analysis result for the analysis image; wherein the at least one artificial intelligence model is pre-trained to infer geometric relationships between a plurality of objects within the image; and A method comprising the step of inputting the above analysis result into at least one subsequent processing component.

2. In Paragraph 1, A method in which at least one artificial intelligence model is pre-trained to infer the geometric relationship through a single inference process.

3. In Paragraph 1, A method comprising at least one of the above analysis result information regarding the geometric order between a plurality of objects within the analysis image and information regarding the shielding relationship between the plurality of objects.

4. In Paragraph 1, A method further comprising the step of the above at least one subsequent processing component manifesting the analysis result through at least one user interface.

5. In Paragraph 4, The step of expressing the above analysis results through at least one user interface is, A method comprising the step of providing, through the user interface, an image in which different visual effects are applied according to depth order to each of the plurality of objects in the analysis image, based on information regarding the geometric relationships between the plurality of objects in the image.

6. In Paragraph 1, The method wherein the image for analysis comprises at least one of an RGB image, an infrared image, a radar image, a single camera image, a stereo camera image, a CT image, an MRI image, an ultrasound image, a satellite image, and a CCTV image.

7. In Paragraph 1, A method in which at least one artificial intelligence model is pre-trained to take features of the image for analysis as input and output a geometric matrix including information regarding the depth of a plurality of objects within the image for analysis or information regarding the occlusion relationship between the plurality of objects.

8. In Paragraph 1, The above at least one artificial intelligence model is pre-trained by a learning method by the above at least one processor, and The above learning method is, A step of generating multiple pixel-wise feature maps from a training image; A step of generating a plurality of mask embeddings by applying a predetermined correlation modeling operation to the plurality of pixel-specific feature maps and a plurality of trainable sub-embeddings; A step of generating a plurality of binary masks through a predetermined operation on the plurality of pixel-specific feature maps and the plurality of mask embeddings; A step of generating multiple object region-specific feature maps by applying the above-mentioned plurality of binary masks to the above-mentioned plurality of pixel-specific feature maps; A step of generating a plurality of object-specific global descriptor embeddings using a predetermined operation that models the interrelationships of the plurality of object-specific feature embeddings; A step of combining a portion corresponding to a plurality of correct answer objects among the plurality of mask embeddings with the plurality of object-specific global descriptor embeddings; A predetermined operation for modeling the interaction between a plurality of embeddings included in the above combination, a step of dividing the plurality of embeddings generated as a result of the modeling into two groups, and generating a predetermined geometric matrix based on the matrix product of the two groups; and A method comprising the step of updating the parameters of at least one artificial intelligence model so as to minimize the loss based on the geometric matrix and the ground truth data.

9. In Paragraph 8, The step of generating the above plurality of binary masks is, A step of calculating a plurality of mask scores by performing a matrix multiplication operation on the plurality of pixel-specific feature maps and the plurality of mask embeddings; A step of generating a plurality of candidate binary masks through non-linear operation and binarization on the plurality of mask scores; and A method comprising the step of selecting a plurality of binary masks corresponding to a plurality of correct answer objects by applying an optimal assignment algorithm to the plurality of candidate binary masks.

10. In Paragraph 8, The step of generating feature maps for each of the above-mentioned multiple object regions is: A method comprising the step of generating a plurality of object region-specific feature maps by performing an element-wise multiplication operation on the plurality of binary masks and the plurality of pixel-specific feature maps.

11. In Paragraph 8, The step of generating the above-mentioned plurality of object-specific global descriptor embeddings is, A step of performing a predetermined operation to model the interrelationships of the plurality of object region-specific feature embeddings; and A method comprising: a step of generating a plurality of object-specific global descriptor embeddings by performing a predetermined pooling operation to compress the dimensions of the above modeling result.

12. In Paragraph 8, Some of the plurality of mask embeddings above that correspond to a plurality of correct answer objects are, A method selected by applying an optimal allocation algorithm to the plurality of mask embeddings and selecting a portion corresponding to the plurality of correct answer objects corresponding to the plurality of correct answer objects.

13. In Paragraph 8, The step of generating the above geometric matrix is, A step of dividing a plurality of embeddings generated as a result of the modeling for the above combination into a first group corresponding to some of the plurality of mask embeddings and a second group corresponding to the plurality of object-specific global descriptor embeddings; A step of obtaining a geometric relation feature representation generated by applying a predetermined non-linear transformation and matrix multiplication operation to each of the first group and the second group; and A method comprising the step of generating the geometric matrix by performing a predetermined non-linear transformation on the geometric relation feature representation above.

14. In Paragraph 1, A method in which at least one artificial intelligence model is pre-trained to calculate semantic importance for a plurality of tokens generated from an image, reconstruct the plurality of tokens by merging some tokens with low importance, and infer geometric relationships between the plurality of entities based on the reconstructed plurality of tokens.

15. In Paragraph 14, The above-mentioned at least one artificial intelligence model is, A step of generating multiple tokens based on an input image; A step of classifying the plurality of tokens into a first group having relatively high importance with respect to the input image and a second group having relatively low importance; A step of arbitrarily classifying a plurality of meaningless tokens included in the second group into a plurality of groups; A step of performing a similarity merging operation considering local consistency among the above-mentioned arbitrarily classified multiple groups to convert multiple meaningless tokens into one or more reconstructed tokens; A step of re-evaluating the semantic importance of the input image for an entire token sequence including a plurality of tokens included in the first group and one or more reconstruction tokens, and rearranging the entire token sequence based on the re-evaluated importance; and A method pre-trained by a learning method comprising the step of removing meaningless tokens with importance lower than a predetermined threshold from the entire rearranged token sequence and learning to infer geometric relationships between the plurality of objects using the remaining plurality of tokens.

16. In Paragraph 15, The step of classifying the plurality of tokens into the first group and the second group is A step of calculating the importance between each of the remaining tokens and an overall image token containing information regarding the entire input image included in the plurality of tokens; and A method comprising the step of classifying tokens whose importance is greater than or equal to a predetermined threshold into the first group according to importance relative to the entire image token, and classifying tokens whose importance is less than the predetermined threshold into the second group.

17. At least one memory; and At least one processor that executes an image analysis method based on geometric information between multiple objects in an image using at least one instruction stored in the memory; The above at least one instruction is, A step of receiving an image for analysis and storing it in at least one memory; A step of loading the analysis image from at least one memory; A step in which at least one processor uses at least one artificial intelligence model that takes the analysis image as input to generate an analysis result for the analysis image; wherein the at least one artificial intelligence model is pre-trained to infer geometric relationships between a plurality of objects within the image; and A system comprising a command to perform the step of ingesting the above analysis result into at least one subsequent processing component.

18. In Paragraph 17, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; and at least one routing network that controls the data flow between the plurality of neurons; comprising A system further comprising a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to establish a transmission path for the weights.

19. In Paragraph 17, A plurality of neurons organized into an array comprising at least one register, at least one microprocessor, and at least one input; and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; comprising A system comprising an Application Specific Integrated Circuit (ASIC) for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through any one of the plurality of synaptic circuits.

Images