Spatial information processing system and method

Abstract

A spatial information processing system according to an embodiment of the present invention may comprise the operations of: receiving an image and a query as inputs; extracting, from the image, a multi-dimensional visual feature representation including captions, object coordinates, and OCR text; converting the multi-dimensional visual feature representation into a linguistic representation understandable by a large-scale language model by using at least one lightweight artificial intelligence model; and receiving the converted linguistic representation and the query as inputs and generating a final response.

Description

Spatial Information Processing System and Method

[0001] The present invention relates to a spatial information processing system and method, and more specifically, to a spatial information processing system and method that links numerical and structural spatial information, such as images, coordinates, and bounding boxes, with natural language representations.

[0002] Large Language Models (LLMs) are artificial intelligence models capable of understanding and generating natural language and other types of content by learning from vast amounts of data. They are language models composed of artificial neural networks containing numerous parameters (typically billions of weights or more). LLMs utilize machine learning techniques to understand and generate human language; they can automate various aspects of communication and data processing and learn complex patterns and relationships within diverse sentences. Recently, LLMs have been applied across various industries by incorporating capabilities such as extracting keywords from text, understanding context and intent, and analyzing context. In short, recent LLMs are demonstrating performance in various natural language processing fields that is close to or even surpasses human capabilities.

[0003] Recently, multimodal research aimed at extending the capabilities of such LLMs to visual information has been actively underway. However, existing LLMs are inherently text-based, lack the ability to directly interpret visual feature representations, and have fundamental limitations in processing spatial information.

[0004] For example, existing LLMs exhibit limitations in that, despite receiving explicit instructions to "convert bounding box coordinates into natural positional representations," they fail to interpret the spatial meaning of the coordinates (e.g., 'left', 'right') and output numerical coordinate values ​​([0.42,..., 0.02]) as is. This is a clear example demonstrating the lack of ability in LLMs to accurately interpret spatial relationships and express them in language. Furthermore, spatial hallucinations—where Vision-Language Models (VLMs) rely on "spurious patterns" that incorrectly link actions unrelated to the actual scene, thereby mentioning non-existent objects or actions—are also emerging as a problem.

[0005] The limitations of such spatial analysis in LLMs are a major obstacle restricting their use in critical application fields where accurate spatial recognition is essential, such as medical image analysis, autonomous driving, and robot control. (Korean Registered Patent No. 10-2745048)

[0006] The main purpose of the present invention is to solve the fundamental limitation of existing LLMs as described above, namely the problem of being unable to convert numerical spatial information (coordinates, bounding boxes, etc.) into semantic natural language (position, relationship).

[0007] In addition, it aims to suppress spatial hallucinations (spurious patterns) that occur regardless of the actual location or attributes of an object, enhance the accuracy of inferring complex spatial relationships, and provide consistent spatial perception even with geometric transformations such as image rotation, enlargement, and reduction.

[0008] In addition, it aims to provide an efficient and modular system and method that enables LLM to understand and represent space in a manner similar to humans.

[0009] A system performed by a computer according to an embodiment of the present invention comprises: at least one memory for storing instructions; and at least one processor for executing instructions to perform an operation. The operation performed by the at least one processor by executing instructions may include: receiving an image and a query; extracting a multidimensional visual feature representation including a caption, object coordinates, and OCR text from the image; converting the multidimensional visual feature representation into a linguistic expression understandable by a large-scale language model using at least one lightweight artificial intelligence model; and receiving the converted linguistic expression and the query to generate a final response.

[0010] Here, the at least one lightweight artificial intelligence model includes a spatial-language mapping neural module (SLMM) separated from the large-scale language model, and the transforming operation may include: an operation of transforming numerical coordinate values ​​included in the multidimensional visual feature representation into a spatial semantic vector containing spatial meaning; and an operation of generating a natural language-based spatial representation report as the linguistic representation by performing cross-attention to map the spatial semantic vector to a corresponding linguistic representation.

[0011] Meanwhile, the above-mentioned transforming operation may include: an operation of decomposing the image into a plurality of image patches and decomposing the query into a plurality of text tokens and combining them into a single fused sequence; an operation of performing self-attention on the entire single fused sequence to learn the interrelationships and context between the image patches and text tokens; and an operation of inferring explicit spatial relationships between objects based on the learned context to generate a structured graph, and converting the structured graph into text to generate the linguistic expression.

[0012] Meanwhile, the transforming operation may include: an operation to generate context information by performing self-attention on a single fused sequence combining an image patch and a text token; an operation to generate an internal spatial knowledge graph that structures the current state of an image based on the context information; an operation to identify entities from the internal spatial knowledge graph and search for and collect external knowledge by referring to a pre-built external spatial knowledge graph database using the entities as queries; and an operation to generate an augmented context by integrating the internal spatial knowledge graph and the collected external knowledge, and to generate the augmented context as the linguistic expression.

[0013] Meanwhile, the operation of extracting the multidimensional visual feature representation includes the operation of generating a spatial token sequence in which a visual feature vector and a spatial information vector are combined from the image; the transforming operation may include the operation of receiving the language token sequence generated from the query and the spatial token sequence as input, and performing bidirectional attention on the spatial token sequence to identify the entire spatial layout of the image; and the operation of generating sequential language meaning by performing causal attention on the language token sequence to mask future tokens so that each token refers only to past tokens that are ahead of it.

[0014] According to an embodiment of the present invention, the problem of existing LLMs failing to convert numerical spatial information (coordinates, bounding boxes, etc.) into semantic natural language (location, relationship) can be resolved. Furthermore, it can suppress spatial spurious patterns that occur regardless of the actual location or attributes of an object, improve the accuracy of complex spatial relationship inference, and provide consistent spatial recognition even with geometric transformations such as rotation, enlargement, and reduction of images. Additionally, it can provide an efficient and modular system and method that enables the LLM to understand and represent space similarly to humans. Moreover, it can maximize the response reliability of the LLM in specialized fields requiring high accuracy, such as medicine and autonomous driving, and efficiently expand functionality without the need to retrain a massive LLM through a modular approach like SLMM.

[0015] FIG. 1 is a schematic diagram of an electronic device according to one embodiment of the present invention.

[0016] FIG. 2 is a schematic diagram of a spatial information processing system according to a first embodiment of the present invention.

[0017] FIG. 3 is a schematic diagram of a spatial information processing system according to a second embodiment of the present invention.

[0018] FIG. 4 is a schematic diagram of a spatial information processing system according to a third embodiment of the present invention.

[0019] FIG. 5 is a schematic diagram of a spatial information processing system according to a fourth embodiment of the present invention.

[0020] FIG. 6 is a flowchart of a spatial information processing method according to one embodiment of the present invention.

[0021] FIG. 7 is a flowchart of a spatial information processing method according to a first embodiment of the present invention.

[0022] FIG. 8 is a flowchart of a spatial information processing method according to a second embodiment of the present invention.

[0023] FIG. 9 is a flowchart of a spatial information processing method according to a third embodiment of the present invention.

[0024] FIG. 10 is a flowchart of a spatial information processing method according to the fourth embodiment of the present invention.

[0025] Figure 11 shows an exemplary on-premises full-stack structure.

[0026] To clarify the technical concept of the present disclosure, embodiments of the present invention will be described in detail with reference to the attached drawings. In describing the present disclosure, detailed descriptions of related known functions or components will be omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the present disclosure. Components having substantially the same functional configuration in the drawings have been assigned the same reference numerals and symbols as much as possible, even if they are shown in different drawings. For convenience of explanation, devices and methods will be described together where necessary. Each operation of the present disclosure does not necessarily need to be performed in the order described and may be performed in parallel, selectively, or individually.

[0027] The terms used in the embodiments of this disclosure have been selected to be as widely used and general as possible, taking into account the functions of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, terms used in this specification should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0028] Throughout this disclosure, singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms such as “comprising” or “having” are intended to specify the presence of features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, throughout this disclosure, when a part is described as “comprising” a certain component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0029] Expressions such as "at least one" modify the entire list of components and do not modify the components of the list individually. For example, "at least one of A, B, and C" and "at least one of A, B, or C" refer to only A, only B, only C, both A and B, both B and C, both A and C, all of A, B, and C, or any combination thereof.

[0030] Additionally, terms such as “...part,” “...module,” etc., as described in this disclosure refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software.

[0031] Throughout the entire disclosure, when a part is described as being “connected” to another part, this includes not only cases where they are “directly connected” but also cases where they are “electrically connected” with other elements interposed between them. Furthermore, when a part is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0032] As used throughout this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0033] Artificial intelligence (AI) is a field of computer science and information technology that studies methods to enable computers to perform thinking, learning, and self-development—tasks achievable by human intelligence—and refers to the ability of computers to mimic intelligent human behavior. Furthermore, AI does not exist in isolation but is closely related, directly or indirectly, to many other fields of computer science. Particularly in the modern era, there are very active attempts to introduce AI elements into various sectors of information technology and utilize them to solve problems within those fields.

[0034] Machine learning is a field of artificial intelligence that enables computers to learn without explicit programming. Specifically, machine learning can be defined as a technology that studies and builds systems and algorithms capable of learning, making predictions, and improving their own performance based on empirical data. Rather than executing strictly defined static program commands, machine learning algorithms adopt an approach of constructing specific models to derive predictions or decisions based on input data.

[0035] Many machine learning algorithms have been developed to address how to classify data in machine learning. Representative examples include Decision Trees, Bayesian Networks, Support Vector Machines (SVMs), and Artificial Neural Networks (ANNs). A Decision Tree is an analytical method that performs classification and prediction by plotting decision rules in a tree structure. A Bayesian Network is a model that represents the probabilistic relationships (conditional independence) between multiple variables in a graph structure. Bayesian Networks are suitable for data mining through unsupervised learning. Support Vector Machines are supervised learning models for pattern recognition and data analysis, primarily used for classification and regression analysis. Artificial Neural Networks model the operating principles of biological neurons and the relationships between them; they are information processing systems in which multiple neurons, referred to as nodes or processing elements, are connected in a layered structure.

[0036] Artificial neural networks are models used in machine learning, serving as statistical learning algorithms in machine learning and cognitive science that draw inspiration from biological neural networks (particularly the brain within the animal central nervous system). Specifically, an artificial neural network can refer to a model in which artificial neurons (nodes), forming a network through the connection of synapses, change the strength of these connections through learning to possess problem-solving capabilities.

[0037] An artificial neural network may include multiple layers, and each layer may include multiple neurons. Additionally, an artificial neural network may include synapses connecting neurons. An artificial neural network can generally be defined by the following three factors: ㉠ connection patterns between neurons of different layers, ㉡ a learning process that updates the weights of the connections, and ㉢ an activation function that generates an output value from a weighted sum of inputs received from the previous layer.

[0038] Artificial neural networks may include, but are not limited to, network models such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), BRDNN (Bidirectional Recurrent Deep Neural Network), MLP (Multilayer Perceptron), and CNN (Convolutional Neural Network).

[0039] Artificial neural networks are classified into single-layer neural networks and multi-layer neural networks depending on the number of layers. A typical single-layer neural network consists of an input layer and an output layer. Additionally, a typical multi-layer neural network consists of an input layer, one or more hidden layers, and an output layer.

[0040] The input layer is a layer that receives external data, and the number of neurons in the input layer is equal to the number of input variables. The hidden layer is located between the input layer and the output layer, receives signals from the input layer, extracts features, and transmits them to the output layer. The output layer receives signals from the hidden layer and outputs an output value based on the received signals. Input signals between neurons are multiplied by their respective connection strengths (weights) and then summed; if this sum is greater than the neuron's threshold, the neuron is activated and outputs the value obtained through the activation function.

[0041] Meanwhile, a deep neural network that includes multiple hidden layers between the input layer and the output layer can be a representative artificial neural network that implements deep learning, a type of machine learning technology.

[0042] The machine learning workflow consists of a series of processes involving collecting data for learning and validation, modeling, and training the model, and may include the processes of collecting training data, checking and exploring data, data preprocessing and cleaning, modeling, and training.

[0043] 1. Collect Training Data

[0044] Training data applied to the training of the learning model of this specification may be generated using data collected from a plurality of samples. In this specification, at least one different type of training data set may be used to train the learning model, and each training data may further include one or more experimental results used as feature labels. At least a portion of the training data set may be used to train the learning model, and another portion may be used to validate the learned learning model.

[0045] 2. Data Inspection and Exploration

[0046] Once training data for training a learning model is collected, the collected training data can be examined and explored regarding its structure, noise data, and data cleaning methods for machine learning applications.

[0047] This stage of data inspection and exploration is called Exploratory Data Analysis (EDA), which can be described as the process of observing and understanding collected data from various angles. Before training the data, independent variables, dependent variables, variable types, and data types are examined using visualizations such as graphs and statistical tests, allowing the characteristics of the data and inherent structural relationships to be identified in advance. Through this EDA, examining the distribution and values ​​of the data enables a better understanding of the phenomena represented by the data and the discovery of potential problems. Furthermore, by examining the data from various angles, diverse patterns that might not have been identified during the problem definition stage can be discovered, allowing for the modification of existing hypotheses or the formulation of new ones. Exploratory data analysis can broadly encompass the process of searching for data outliers and analyzing the relationships between data attributes.

[0048] The process of detecting outliers involves verifying whether the data contains them and can include sampling, statistical, and visualization methods. Sampling methods involve drawing random samples from the data to identify overall trends and anomalies in the data values. Statistical methods may utilize summary statistics, such as the mean, median, and mode to identify the center of the data, or range and variance to check the dispersion. Visualization methods utilize probability density functions, histograms, dot plots, word clouds, time series charts, and maps to determine which statistical indicators are appropriate for the individual attributes of the collected data. However, when using statistical indicators, caution should be exercised regarding the use of statistical indicators: while the mean reflects all data values ​​within a set and is therefore affected by outliers, the median uses only the single value in the middle, allowing for representative results even in the presence of outliers.

[0049] The process of analyzing relationships between data attributes involves identifying combinations of attributes within the data that possess meaningful correlations. Relationship analysis can be conducted differently depending on the combination of attributes between qualitative attributes (Categorical Variables; Qualitative), which cannot be expressed numerically but can be arbitrarily quantified, and quantitative attributes (Numeric Variables; Quantitative), which can be quantified. Categorical-categorical relationships can display the number of values ​​corresponding to each pair of attribute values ​​using cross-tabulation tables or mosaic plots; Numeric-categorical relationships can be visually represented through box plots or by observing statistical values ​​by category (mean, median, etc.); and Numeric-numeric relationships can analyze the association between two attributes using correlation coefficients. It can be confirmed that a correlation coefficient of -1 indicates a negative correlation where the two attributes change in opposite directions, 0 indicates no correlation, and 1 indicates a positive correlation where the two attributes always change in the same direction. The relationship between two attributes with a correlation coefficient can also exhibit various aspects, which can be visually represented using a scatter plot.

[0050] 3. Data Preprocessing and Cleansing

[0051] Data that has completed inspection and exploration undergoes data preprocessing to transform it into a format suitable for machine learning training models. Data preprocessing involves cleaning the data and converting it into a form that the model can understand; it generally includes handling missing data, outlier removal, data scaling, categorical data encoding, feature selection and extraction, and data transformation. The detailed processes of data preprocessing may be performed in whole or in part selectively, and a separate machine learning model may be used for this purpose.

[0052] Handling Missing Data is the process of handling missing values ​​when they exist in the data; these values ​​can be displayed as NaN (Not a Number) or empty, or deleted. Filling in or deleting missing values ​​improves data completeness, and values ​​such as the mean, median, or mode may be used when filling in missing values.

[0053] Outlier removal is the process of eliminating outliers, which are values ​​that deviate from typical data patterns. Since outliers can degrade model performance, they must be removed or replaced; this involves identifying outliers and deleting the corresponding rows or columns or replacing them with other values.

[0054] Data scaling is the process of adjusting the size of data; through data scaling, the range of the data is adjusted, which can improve model performance or accelerate convergence. Data scaling allows data characteristics to be aligned within a similar range, and generally, standardization and normalization can be applied. Standardization is a method of transforming data into a distribution with a mean of 0 and a standard deviation of 1; it is primarily performed using the mean and standard deviation, and the standardized value z is It can be denoted as (where x is the original value, μ is the mean, and σ is the standard deviation). Normalization is a method of transforming the range of data to [0,1] or [-1,1], primarily using minimum and maximum values ​​to transform the data, and the normalized value x norm silver It can be expressed as (x is the original value, x min is the minimum value, x max is the maximum value).

[0055] Categorical Data Encoding is the process of converting categorical variables, which are represented as string or integer values ​​and cannot be directly input into a model, into a numeric type that can be input. Generally, one-hot encoding or label encoding can be used to convert categorical variables into numeric types.

[0056] Feature selection and extraction is intended to improve the performance of a model by selecting the most useful features for model training or extracting new features. Through this process, the complexity of the model can be reduced and overfitting can be prevented.

[0057] Data transformation involves converting data to extract new information or enable a model to understand it better, and may include the tokenization of text data or the preprocessing of image data. Through data transformation, model performance can be improved by extracting useful features from original data or converting data into an appropriate format.

[0058] Through data preprocessing as described above, it is possible to achieve the effects of improving the performance and ensuring the stability of machine learning models.

[0059] Meanwhile, when training a learning model according to one embodiment of the present invention, a process of preprocessing information written in natural language and a process of training a large-scale language model based on the preprocessed data may be performed.

[0060] 3-1. Text Preprocessing for Large-Scale Language Models

[0061] If the collected data has not been preprocessed according to the requirements, tokenization, cleaning, and normalization can be performed to suit the intended use of the data.

[0062] Tokenization refers to the process of dividing given data into units called tokens, and these token units can generally be defined as meaningful units. Tokenization can broadly include word tokenization and sentence tokenization.

[0063] Tokenization refers to the process of dividing given data into units called tokens, and these token units can generally be defined as meaningful units. Tokenization can broadly include word tokenization and sentence tokenization.

[0064] Word tokenization refers to the case where the standard for tokens is a word; here, a word can include not only individual words but also phrases or meaningful strings. Word tokenization means separating words based on spaces or punctuation marks, such as periods, commas, question marks, semicolons, and exclamation marks. However, since removing all punctuation or special characters during the tokenization process can cause tokens to lose their meaning, precise algorithms may be required. For instance, if a word itself contains punctuation or uses special characters with meaning, simply removing them may not be sufficient. Therefore, tokenization rules such as Penn Treebank Tokenization rules may be applied during the process.

[0065] Sentence tokenization refers to the process of dividing text into sentence units. Typically, if data is unrefined, the corpus is not organized into sentences, so sentence tokenization may be necessary to suit the intended use. Various rules for this sentence tokenization can be defined depending on the language used and how special characters are utilized within the corpus.

[0066] The process of classifying tokens according to their purpose is called tokenization, and before and after tokenization, text data undergoes cleaning and normalization tailored to its intended use. Cleaning involves removing noise data, while normalization involves consolidating words with different representations into a single word.

[0067] Cleansing is sometimes performed prior to tokenization to exclude elements that interfere with the process, but it can also be repeatedly carried out after tokenization to remove noise that remains. The noise data removed during cleansing consists of meaningless characters; methods for eliminating unnecessary words include stopword removal, as well as removing infrequent and short words.

[0068] Normalization tasks include the consolidation of words with different spellings and case consolidation based on rules. Case consolidation is a normalization method that can reduce the number of words in English-speaking languages; since uppercase letters are used only in specific situations, such as at the beginning of a sentence, and most text is written in lowercase, case consolidation can mostly be accomplished through the conversion of uppercase letters to lowercase.

[0069] To process natural language in computing systems, a preprocessing step of converting text into numerical values ​​is required; for this purpose, each word in the text is mapped to a unique integer. This mapping process can utilize techniques such as integer encoding, padding, and one-hot encoding.

[0070] Integer encoding is a method of assigning integers to words. It involves creating a vocabulary sorted by frequency and assigning integers sequentially from lowest to highest frequency. Integer encoding performs sentence tokenization on text data containing multiple sentences, and simultaneously conducts word tokenization through cleaning and normalization processes. During this process, words are converted to lowercase to standardize the word count, and words can be removed based on stop words or word length. Through this, words can be recorded as keys and their frequencies as values. Integer encoding is performed by sorting words within the text in order of frequency and assigning integers to the words with the highest frequencies.

[0071] Padding is a process used to arbitrarily equalize the lengths of sentences of different lengths within a text. Computing systems can perform parallel operations by grouping sentences of equal length into a single matrix. In other words, to perform parallel operations, the lengths of sentences can be equalized by arbitrarily filling the integer encoding results of sentences of different lengths within the text with '0's. That is, the longest sentence is identified from the set of integer-encoded words, and "0"s can be added to the integer matrix corresponding to the length of that longest sentence. The computing system can proceed with parallel processing by recognizing sentences of equal length as a single matrix, and in this process, the "0" words, which are perceived as meaningless, can be ignored. Adjusting the size (shape) of data by filling it with specific values ​​in this manner is called padding, and when the number "0" is used to adjust length, it is referred to as zero padding.

[0072] One-hot encoding is a vector representation method in which the size of the word set is used as the dimension of the vector, and a value of 1 is assigned to the index of the word to be represented and 0 to other indices; the vector represented in this way is called a one-hot vector. One-hot encoding consists of integer encoding and index assignment processes. After integer encoding is performed to assign a unique integer to each word, the unique integer of the word to be represented is considered as the index, and a "1" is assigned to that position, while a "0" is assigned to the index positions of other words. However, one-hot encoding has the disadvantage that the space required to store the vector increases (increase in vector dimension) as the number of words increases, and it is also impossible to verify similarity between words. To address these drawbacks, techniques that vectorize in a multi-dimensional space by reflecting the latent meaning of words are available. These include count-based vectorization methods such as LSA (Latent Semantic Analysis); prediction-based vectorization methods such as NNLM, RNNLM, Word2Vec, and FastText; and the GloVe method, which uses both count-based and prediction-based approaches.

[0073] Meanwhile, in order for a computer to understand and process text, it must be appropriately converted into numbers. Since the performance of natural language processing varies significantly depending on how words are represented, many techniques have been proposed to quantify words. Currently, word embedding, which vectorizes each word through artificial neural network learning, is the most widely used method.

[0074] Word embedding is a method of representing words as vectors, converting words into dense representations. The result derived through the word embedding process is called a dense vector or embedding vector. Word embedding methodologies include LSA, Word2Vec, FastText, and Glove.

[0075] 4. Modeling and Training

[0076] Artificial neural networks can be trained using training data. Here, training refers to the process of determining the parameters of an artificial neural network using training data to achieve objectives such as classifying, regressing, or clustering input data. Typical examples of artificial neural network parameters include weights assigned to synapses or biases applied to neurons.

[0077] An artificial neural network trained on training data can classify or cluster input data according to the patterns of the input data. Meanwhile, an artificial neural network trained using training data may be referred to as a trained model in this specification.

[0078] The following explains the learning methods of artificial neural networks. The learning methods of artificial neural networks can be broadly classified into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0079] Supervised learning is a method of machine learning designed to infer a function from training data. Among the functions inferred in this way, outputting a continuous value is called regression, and predicting and outputting the class of an input vector is called classification.

[0080] In supervised learning, an artificial neural network is trained with labels for the training data. Here, a label refers to the correct answer (or result value) that the artificial neural network must infer when training data is input into the artificial neural network. In this specification, the correct answer (or result value) that the artificial neural network must infer when training data is input is referred to as a label or labeling data. Furthermore, in this specification, setting labels for the training data to train the artificial neural network is referred to as labeling the training data. In this case, the training data and the corresponding labels constitute a single training set, and can be input to the artificial neural network in the form of a training set.

[0081] Meanwhile, training data represents multiple features, and labeling the training data implies that labels are attached to the features represented by the training data. In this case, the training data can represent the features of the input object in the form of a vector. An artificial neural network can infer a function regarding the association between the training data and the labeled data by utilizing the training data and the labeled data. Furthermore, the parameters of the artificial neural network can be determined (optimized) through the evaluation of the function inferred by the network.

[0082] Unsupervised learning is a type of machine learning in which no labels are provided for the training data. Specifically, unsupervised learning can be a learning method that trains an artificial neural network to find and classify patterns within the training data itself, rather than focusing on the relationship between the training data and its corresponding labels. Examples of unsupervised learning include clustering and Independent Component Analysis.

[0083] Examples of artificial neural networks that utilize unsupervised learning include Generative Adversarial Networks (GANs) and Autoencoders (AEs).

[0084] Generative Adversarial Networks (GANs) are machine learning methods in which two distinct artificial intelligence models—a generator and a discriminator—compete to improve performance. In this context, the generator is a model that creates new data, capable of generating new data based on original data. The discriminator, on the other hand, is a model that recognizes data patterns, performing the role of distinguishing whether input data is original data or new data generated by the generator. Furthermore, the generator learns by receiving input data that failed to deceive the discriminator, while the discriminator learns by receiving input data that was deceived by the generator. Consequently, the generator can evolve to deceive the discriminator as effectively as possible, and the discriminator can evolve to better distinguish between original data and data generated by the generator.

[0085] An autoencoder is a neural network that aims to reproduce the input itself as the output. An autoencoder includes an input layer, at least one hidden layer, and an output layer. In this case, since the number of nodes in the hidden layer is less than the number of nodes in the input layer, the dimensionality of the data is reduced, and accordingly, compression or encoding is performed. Additionally, the data output from the hidden layer enters the output layer. In this case, since the number of nodes in the output layer is greater than the number of nodes in the hidden layer, the dimensionality of the data is increased, and accordingly, decompression or decoding is performed.

[0086] Meanwhile, an autoencoder represents input data as hidden layer data by adjusting the connection strengths of neurons through learning. In the hidden layer, information is represented with fewer neurons than in the input layer, and the fact that input data can be reproduced as output implies that the hidden layer has discovered and represented hidden patterns from the input data.

[0087] Semi-supervised learning is a type of machine learning that refers to a learning method utilizing both labeled and unlabeled training data. One technique within semi-supervised learning involves inferring labels from unlabeled training data and then performing learning using those inferred labels; this method can be particularly useful when the cost of labeling is high.

[0088] Reinforcement learning is a theory that states that if an agent is provided with an environment where it can determine the best action to take at every moment, it can find the optimal path through experience alone, without relying on data. Reinforcement learning is primarily executed via a Markov Decision Process (MDP). To explain the MDP, first, an environment is provided containing the information necessary for the agent to take its next action; second, the agent's behavior within that environment is defined; third, rewards are determined for success and penalties for failure; and fourth, the optimal policy is derived through repeated experience until future rewards reach their peak.

[0089] The structure of an artificial neural network is determined by the configuration of the model, activation function, loss function or cost function, learning algorithm, optimization algorithm, etc., and hyperparameters are set in advance before learning, and model parameters are set through learning thereafter, so the content can be determined.

[0090] For example, factors determining the structure of an artificial neural network may include the number of hidden layers, the number of hidden nodes included in each hidden layer, the input feature vector, the target feature vector, etc.

[0091] Hyperparameters include various parameters that must be initially set for training, such as the initial values ​​of model parameters. Model parameters, on the other hand, include various parameters intended to be determined through training. For example, hyperparameters may include initial values ​​for inter-node weights, initial values ​​for inter-node bias, mini-batch size, number of training iterations, and learning rate. Additionally, model parameters may include inter-node weights and inter-node bias.

[0092] A loss function can be used as an indicator (criterion) to determine optimal model parameters during the learning process of an artificial neural network. In an artificial neural network, learning refers to the process of manipulating model parameters to reduce the loss function, and the objective of learning can be viewed as determining model parameters that minimize the loss function. The loss function can primarily be the Mean Squared Error (MSE) or the Cross Entropy Error (CEE), but the present invention is not limited thereto. The Cross Entropy Error can be used when the correct label is one-hot encoded. One-hot encoding is an encoding method in which the correct label value is set to 1 only for neurons corresponding to the correct answer, and the correct label value is set to 0 for neurons that are not the correct answer.

[0093] In machine learning or deep learning, learning optimization algorithms can be used to minimize the loss function, and learning optimization algorithms include Gradient Descent (GD), Stochastic Gradient Descent (SGD), Momentum, Nesterov Accelerate Gradient (NAG), Adagrad, AdaDelta, RMSProp, Adam, Nadam, etc.

[0094] Gradient Descent is a technique that adjusts model parameters in a direction that reduces the loss function value by considering the gradient of the loss function from the current state. The direction in which model parameters are adjusted is called the step direction, and the magnitude of the adjustment is called the step size. In this context, the step size can refer to the learning rate. Gradient Descent obtains the gradient by taking the partial derivative of the loss function with respect to each model parameter, and updates the model parameters by changing them in the direction of the obtained gradient by the learning rate.

[0095] Stochastic Gradient Descent is a technique that divides training data into mini-batches and performs gradient descent on each mini-batch to increase the frequency of gradient descent.

[0096] Adagrad, AdaDelta, and RMSProp are techniques that improve optimization accuracy in SGD by adjusting the step size. In SGD, Momentum and NAG are techniques that improve optimization accuracy by adjusting the step direction. Adam is a technique that improves optimization accuracy by combining Momentum and RMSProp to adjust both the step size and the step direction. Nadam is a technique that improves optimization accuracy by combining NAG and RMSProp to adjust both the step size and the step direction.

[0097] The learning speed and accuracy of artificial neural networks are characterized by being heavily dependent on hyperparameters, as well as the network structure and the type of learning optimization algorithm. Therefore, to obtain a good learning model, it is important to set appropriate hyperparameters in addition to determining a suitable network structure and learning algorithm.

[0098] Typically, hyperparameters are experimentally set to various values ​​while training the artificial neural network, and then set to the optimal value that provides stable training speed and accuracy based on the training results.

[0099] FIG. 1 is a schematic diagram of an electronic device according to one embodiment of the present invention, FIG. 2 is a schematic diagram of a spatial information processing system according to one embodiment of the present invention, FIG. 3 is a schematic diagram of a spatial information processing system according to one embodiment of the present invention, FIG. 4 is a schematic diagram of a spatial information processing system according to one embodiment of the present invention, FIG. 5 is a schematic diagram of a spatial information processing system according to one embodiment of the present invention, and FIG. 6 is a flowchart of a spatial information processing method according to one embodiment of the present invention. FIG. 7 to FIG. 10 are flowcharts of spatial information processing methods according to one embodiment to one embodiment of the present invention. FIG. 11 is an exemplary on-premise full-stack structure.

[0100] As illustrated in FIG. 1, an electronic device (100) according to embodiments of the present invention may include a processor (110), memory (120), a communication unit (130), and an input / output unit (140). The electronic device (100) is a basic configuration for performing a computing environment, and the electronic device (100) may be implemented with some other components additionally or substantially in other embodiments, implemented as a single or multiple entity, or implemented as only some of the disclosed configurations. Internal or external components of the electronic device (100), or at least some of the components, may transmit or receive data or signals by being connected to each other through a BUS, GPIO (General Purpose Input / Output), SPI (Serial Peripheral Interface), or MIPI (Mobile Industry Processor Interface), etc.

[0101] The processor (110) may mean a set of one or more processors unless the context clearly indicates otherwise, and can control components of the processor (110) and the electronic device (100) by running software (e.g., instructions, programs, etc.) stored in memory (120). Additionally, the processor (110) can perform various operations such as computation, processing, data generation or processing, and can read data from memory (120) or store it in memory (120). The processor (110) may be composed of at least one core and may include a processor for data analysis, machine learning (ML), or deep learning (DL), such as a Central Processing Unit (CPU), a General Purpose Graphics Processing Unit (GPGPU), or a Tensor Processing Unit (TPU). The processor (110) can read software stored in memory (120) and perform data processing for machine learning (or deep learning) according to the present invention. According to one embodiment of the present disclosure, the processor (110) can perform operations for training a neural network. The processor (110) can perform operations for training a neural network, such as processing input data for training in deep learning, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) can process the training of the neural network model. For example, the CPU and GPGPU can together process the training of the neural network model and data classification using the neural network model. In addition, in one embodiment of the present disclosure, at least one processor (110) of the electronic device (100) can be used together to process the training of the neural network model and data classification using the neural network model.

[0102] Memory (120) is intended to store various data, and the data may include software (e.g., instructions, programs, etc.) which is acquired, processed, or used by at least one component of the electronic device (100). Unless otherwise clearly expressed in the context, memory (120) may mean a set of one or more memories and may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM (Static Random Access Memory), ROM, EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, optical disk, and web storage that performs storage functions on the internet. The instruction, program, or software stored in the memory (120) may be used to refer to an operating system, an application, or middleware that provides various functions to an application to enable the application to utilize the components of the electronic device (100) for controlling the components of the electronic device (100). In one embodiment, when the processor (110) performs a specific operation, the memory (120) may store instructions that are performed by the processor (110) and correspond to the specific operation.

[0103] The communication unit (130) performs wireless or wired communication between the electronic device (100) and another device (e.g., a user terminal or another server), and the communication unit (130) may use wireless communication systems according to methods such as eMBB, URLLC, MMTC, LTE, LTE-A, NR, UMTS, GSM, CDMA, WCDMA, TDMA, FDMA, OFDMA, SCFDMA, WiBro, WiFi, Bluetooth, NFC, GPS, or GNSS. In addition, the communication unit (130) can use various wired communication systems such as USB, HDMI, RS-232 (Recommended Standard-232), POTS (Plain Old Telephone Service), Public Switched Telephone Network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and Local Area Network (LAN). In one embodiment of the present invention, the communication unit (130) can be configured regardless of the mode of communication, such as wired or wireless, and can be configured with various communication networks such as a Personal Area Network (PAN) and a Wide Area Network (WAN). In addition, the communication network may be the known World Wide Web (WWW), and may also utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.

[0104] An electronic device (100) of one embodiment of the present invention may further include an input / output unit (140). The input / output unit (140) may be configured to be divided into an input unit and an output unit, but alternatively, the input / output unit (140) may have an integrated configuration. The input unit may be configured in various ways as a means of data input. For example, the input unit may be configured to receive user input. The input unit may be configured to receive user input from a user terminal. Here, "receiving input" may mean receiving an input signal (or selection signal) corresponding to the user's input based on input made by the user through the input unit configuration provided in the user terminal. The input unit may also be named a user interface module. The input unit may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input unit. Furthermore, in the present invention, the input unit does not necessarily refer to a hardware means, but can be understood as a channel (AI chatbot, prompt) for receiving input from a user. Here, user input may include documents, text, images (or videos), etc. Next, the output unit may output information through an output unit configuration (e.g., a display unit, a touch screen, etc.) provided in a user terminal or computing device. The output unit does not necessarily refer to a hardware means, but can be understood as a channel for outputting results to the user.

[0105] An electronic device (100) according to an embodiment of the present invention may execute software that configures a spatial information processing system or implements a spatial information processing method.

[0106] A spatial information processing system according to embodiments of the present invention may be configured to include, in common, an input unit (10) that receives an image and a query, a vision encoder (30) that extracts a multidimensional visual feature representation (caption, object coordinates, OCR text, etc.) from the image, a lightweight language model block (260, 360, 460, 560) that converts the visual feature representation into a linguistic representation that can be understood by a large-scale language model (40), a large-scale language model (40) that receives the converted linguistic representation and the user's query and generates a final response, and an answer unit (50) that outputs the answer to the user.

[0107] In addition, the spatial information processing system according to the embodiments of the present invention may apply a spatial transformation network (20) (STN; Spatial Transformation Network) to the front end of the vision encoder (30) to increase robustness so as to consistently produce accurate results without responding to unexpected changes, such as when an input image is rotated, enlarged, or tilted. The STN can detect when an input image is rotated or enlarged, estimate transformation parameters, and computationally inversely transform (normalize) the image. Since the vision encoder (30) processes this normalized feature map, the system can always receive consistent spatial information. The STN can be implemented through the steps of image deformation detection, inverse transformation calculation, and image normalization. In deformation detection, when an image of a 'person rotated 90 degrees' is input, the STN can identify key feature points of the image and predict transformation parameters indicating how it has been transformed compared to the original image (training data). For example, parameters (matrix) such as "This image has been rotated 90 degrees clockwise and magnified 1.2 times" can be derived. Next, in the inverse transform calculation, based on the predicted parameters (e.g., "rotated 90 degrees"), a transformation to return the image to its original state (i.e., "rotated 90 degrees counterclockwise") is calculated, and a sampling grid for performing the inverse transform can be generated. In image normalization, the STN can inversely sample pixels from the original "rotated 90 degrees" image using the sampling grid, and the resulting output passing through the STN can be normalized into an image of a person standing upright.

[0108] With reference to FIG. 6, a spatial information processing method performed in a spatial information processing system according to an embodiment of the present invention having the configuration as described above may include the steps of: receiving an image and a query (S10); extracting a multidimensional visual feature representation including a caption, object coordinates, and OCR text from the image (S20); converting the multidimensional visual feature representation into a linguistic representation understandable by a large-scale language model using at least one lightweight artificial intelligence model (S30); and receiving the converted linguistic representation and the query to generate a final response (S40).

[0109] As illustrated in FIG. 2, in a spatial information processing system (200) according to the first embodiment of the present invention, the lightweight language model block (260) can be implemented as a Spatial-Language Mapping Neural Module (SLMM) separated from the LLM.

[0110] In the first embodiment, the lightweight language model block (260) allows the lightweight Small Language Model (SLM) to act as an intermediate translator without modifying the body of the large language model (40) (LLM).

[0111] In this embodiment, image input and query input (e.g., "Please describe the image in detail") are input from a user through the input unit (10), and the image can be processed by the vision encoder (30).

[0112] The vision encoder (30) may be composed of a set of several specialized processing modules rather than a single processing module. In embodiments of the present invention, the vision encoder (30) may extract and convert various information from an input image, and the vision encoder (30) may be configured to include a Captioning Module (31), an Object Detection Module (32), a Dense Region Captioning Module (33), an OCR Module (34), and the like. The detailed configuration of the vision encoder (30) may be implemented in various modified ways.

[0113] The caption module (31) is for generating one or more summary sentences for the entire image, for example, by recognizing the image and generating a full summary text (caption) of the image such as “two men are standing on the grass”.

[0114] The object detection module (32) can recognize an object within an image and derive an object name indicating what the recognized object is and the object's location (coordinates) within the image. That is, the object detection module (32) can detect an object and output the coordinate values ​​of the bounding box of each object. For example, the object detection module (32) can recognize a person within an image to generate an object name "person" and assign an object location to that object to generate object location information such as "person: [0.42, 0.02, 1.0, 0.99]". Meanwhile, the coordinates representing the location of the object may be normalized coordinates set based on the border of the image. For example, person: [0.42, 0.02, 1.0, 0.99] has the format [x1, y1, x2, y2] and may represent the location within the image of the bounding box surrounding the detected object, person. In [x1, y1, x2, y2], x1 and y1 represent the x and y coordinates of the top-left corner of the bounding box, and x2 and y2 represent the x and y coordinates of the bottom-right corner of the box. For example, in a bounding box defined as [0.42, 0.02, 1.0, 0.99], x1 (0.42) may mean that the top-left corner of the bounding box of the detected object is located at 42% of the total width of the image, and y1 (0.02) may mean that the top-left corner of the bounding box is located at 2% of the total height of the image. Here, all coordinate values ​​can be expressed as floating-point numbers between 0 and 1.

[0115] The detailed area caption module (33) is intended to provide a caption for a situational description of a specific area and can serve as an intermediate role between the caption module (31) and the object detection module (32). That is, the detailed area caption module (33) can set a specific area within the image, for example, [0.11, 0.06, 0.51, 0.93], and generate a detailed situational description for that specific area, such as "a man wearing a green Adidas jersey and pink socks is at the rugby stadium." That is, the detailed area caption module (33) can function as a module that explores the main part of the image or multiple areas in detail to specifically describe 'who / what', 'where', and 'what is doing' in each area.

[0116] The OCR module (34) is intended to detect areas of characters included in an image and extract each character. The OCR module (34) may perform preprocessing steps such as color analysis, grayscale conversion, and brightness adjustment of the image to enable clearer recognition of characters within the image. That is, the OCR module (34) can extract text such as "adidas" included in the image.

[0117] The multidimensional visual feature representation generated in this way can be transmitted to a lightweight language model block (260). The lightweight language model block (260) of the first embodiment is implemented as a spatial-language mapping neural module (SLMM), and the SLMM can perform the function of processing coordinate values ​​that the LLM does not understand.

[0118] In other words, SLMM can implement a method of integrating multiple output values, such as multidimensional visual feature representations, into natural language representations by utilizing language models such as GPT. To achieve this integration into natural language representations, SLMM can receive the following instructions. Use the provided information (captions and object data) to create a focused description of the image. Translate relative coordinates into general locations using terms such as "left, right," "center," "upper," "middle," and "lower," emphasizing how objects relate to the entire scene and to each other. Under no circumstances should you include or mention specific bounding box coordinates in the narrative. Use only general spatial terms to describe object locations and relationships. Integrate object data, bounding box information, and captions into a consistent narrative, and use captions to enhance and clarify object locations; however, prioritize object data in the event of inconsistencies (class, count, or location). Use the provided captions as a reference to enhance and clarify the description of each object's location. Do not, under any circumstances, include or mention specific bounding box coordinates in your narrative.Use only general spatial terms to describe object locations and relationships. Integrate object data, bounding box information, and captions into a cohesive narrative, using captions to enhance and clarify object locations, but prioritizing object data if discrepancies arise (in classes, counts, or locations). Use the provided captions as a reference to enhance and clarify the description of each object's location.)".

[0119] In this embodiment, the SLMM may include a spatial information encoder (261) and a spatial-language attention mechanism (262).

[0120] The spatial information encoder (261) is used to convert spatial information, such as images, coordinates, and bounding boxes, into high-dimensional vectors and can be composed of a multi-layer perceptron (MLP). The spatial information encoder (261) receives numerical coordinate values ​​such as [0.42, 0.02, 1.0, 0.99] and can project them into a spatial semantic vector of the same dimension as the embedding space (e.g., 768 dimensions). The vector thus converted numerically contains the spatial meaning of 'right region'.

[0121] The spatial-language attention mechanism (262) can perform cross-attention to connect two different modalities, 'language' and 'space'. For example, the spatial-language attention mechanism (262) can map the two pieces of information by calculating the highest similarity score between the linguistic expression (Query) "right" and the 'right region' spatial vector (Key).

[0122] Finally, the SLMM generates a natural language-based spatial representation report, such as "person1 is on the right side of the image," through this translation and fusion process, and this report can be delivered to a large-scale language model (40) (LLM) along with the user's original query. Through this, the large-scale language model (40) can generate an accurate and rich final answer without spatial illusions based on a natural language-based spatial representation report that is perfectly understandable to the language model, rather than coordinate values.

[0123] With reference to FIG. 7, a spatial information processing method performed in a spatial information processing system (200) according to a first embodiment of the present invention having the configuration as described above may include: a step of receiving an image and a query (S210); a step of extracting a multidimensional visual feature representation from the image (S220); a step of converting numerical coordinate values ​​included in the multidimensional visual feature representation into a spatial semantic vector containing spatial meaning (S230); a step of generating a natural language-based spatial representation report by performing cross-attention mapping the spatial semantic vector to a corresponding linguistic representation (S240); a step of generating a final response by receiving the natural language-based spatial representation report and the query (S250); and a step of outputting the final response (S260).

[0124] As illustrated in FIG. 3, in a spatial information processing system (300) according to a second embodiment of the present invention, a lightweight language model block (360) can be implemented as a multimodal context encoder. In this embodiment, the multimodal context encoder can be implemented in a manner that initially fuses vision information (image) and language information (text query) in the encoding step itself.

[0125] In this embodiment, when an image input and a query input (e.g., "Describe the image in detail") are input from a user through the input unit (10), the image is normalized in the spatial transformation network (20), and the system can immediately combine these two inputs (image, query) in the vision-language integration embedding stage.

[0126] To this end, the lightweight language model block (360) of the second embodiment may be configured to include a vision-language integration embedding unit (361), a context attention unit (362), and a spatial relationship inference network (363).

[0127] The vision-language integration embedding unit (361) can convert and combine different forms of data, such as images (2D pixels) and queries (1D text), into a single format (1D sequence) that can be processed by a transformer (language model).

[0128] Specifically, when a 2D image is input, the vision-language integrated embedding unit (361) can process the image by dividing it into patches, which are image fragments of a certain size, for example, 16x16 pixels, instead of interpreting the image as ‘cat’, ‘person’, etc. (ViT; Vision Transformer method). In addition, the vision-language integrated embedding unit (361) can break down a user’s text query into multiple text tokens. For example, a text query such as “Who is wearing a blue shirt?” can be broken down into [blue], [shirt], [wearing], [person] (tokenization). Afterward, the vision-language integrated embedding unit (361) can combine these two types of data into one long sequence. At this time, a single fusion sequence such as ([CLS], [PATCH_1], [PATCH_2], ..., [PATCH_N], [SEP], [QUERY_TOKEN_1], [QUERY_TOKEN_2], ...) can be generated by using various special tokens, for example, [CLS], [SEP].

[0129] The context attention unit (362) receives the entire generated single fusion sequence as input and can learn the interrelationships and context between image fragments (patches) and text fragments (tokens). The context attention unit (362) can perform the self-attention mechanism of the Transformer model on the entire fused sequence.

[0130] When the context attention unit (362) performs attention on the entire fused sequence, all elements within the sequence may come to refer to each other. For example, the model can learn on its own that the word 'blue' ([Token 1]) modifies 'shirt' ([Token 2]), and that this concept of 'blue shirt' has the highest association with the areas [Patch 7], [Patch 8], and [Patch 15] of the image. In other words, fine relationships that ground text to specific areas of the image can be identified.

[0131] The spatial relationship inference network (363) can infer explicit spatial relationships between objects based on objects and attributes identified by the context attention unit (362). For example, the spatial relationship inference network (363) can infer explicit spatial relationships between objects, such as 'to the left of' or 'above', based on "'blue shirt' is in [Patch 7]". To infer these spatial relationships between objects, the spatial relationship inference network (363) can be composed of a graph neural network (GNN). That is, the spatial relationship inference network (363) can generate a structured graph by configuring objects as nodes and relationships as edges. For example, a structured graph such as (person1)-[wears] -> (hat), (person1)-[left_of] -> (person2)) can be generated.

[0132] Finally, the structured graph generated in this way is converted into text such as "person1 wears hat. person1 is left of person2." and transmitted to a large-scale language model (40) (LLM), and the LLM (40) can generate a final answer based on this structured information and output it to the user through the answer unit (50).

[0133] With reference to FIG. 8, a spatial information processing method performed in a spatial information processing system (300) according to a second embodiment of the present invention having the configuration as described above may include: a step (S310) of decomposing an input image into a plurality of image patches and decomposing a text query into a plurality of text tokens and combining them into a single fusion sequence; a step (S320) of performing a self-attention mechanism on the entire single fusion sequence to learn the mutual relationship and context between the tokens and specific image patches; a step (S330) of inferring explicit spatial relationships between objects based on the learned context to generate a structured graph; a step (S340) of converting the structured graph into text and receiving it as input to generate a final response; and a step (S350) of outputting the final response.

[0134] As illustrated in FIG. 4, in a spatial information processing system (400) according to the third embodiment of the present invention, a lightweight language model block (460) can be implemented in such a way that it dynamically generates an internal spatial knowledge graph based on a multimodal context encoder and augments a response by referencing an external spatial knowledge graph using a core entity of the graph.

[0135] In this embodiment, when an image input (e.g., a photo of Einstein sitting on a café terrace in Paris) and a query input (e.g., "Who is this person, and what do you think influenced this place?") are input from a user through the input unit (10), the image is normalized in the spatial transformation network (20), and the system can immediately combine these two inputs (image, query) in the vision-language integration embedding stage.

[0136] To this end, the lightweight language model block (460) of the third embodiment may be configured to include a vision-language integration embedding unit (461), a context attention unit (462), a spatial relationship inference network (463), a knowledge extraction unit (464), and a context integration unit (465). Here, the vision-language integration embedding unit (461), the context attention unit (462), and the spatial relationship inference network (463) have substantially the same configuration as those of the second embodiment described above, so a detailed description is omitted.

[0137] The vision-language integration embedding unit (461) can decompose an input image into multiple patches and a query into multiple tokens, and then generate a single long fusion sequence. The sequence generated in this way is input into the context attention unit (462) so that the relationship between the patches and tokens can be learned. The vision-language integration embedding unit (461) can perform self-attention on the fused sequence to generate context information indicating the contextual association between the patches and tokens (words). For example, [Einstein patch] and [person token] can be contextually connected.

[0138] The spatial relationship inference network (463) can generate an internal spatial knowledge graph (SKG) based on this context. The spatial relationship inference network (463) receives basic context information generated by the context attention unit (462) and can construct a dynamic internal spatial knowledge graph (or scene graph) based only on the input image and query. That is, the spatial relationship inference network (463) can generate an SKG by utilizing a Graph Neural Network (GNN) to explicitly extract objects (Nodes) and relationships (Edges) from the basic context information. For example, the internal SKG can structure the current state of the image, such as (Object: Einstein)-[Relationship: Sitting] -> (Object: Chair), (Object: Chair)-[Relationship: Located] -> (Object: Terrace).

[0139] The knowledge extraction unit (464) can search for and collect related knowledge by referring to a pre-built external spatial knowledge graph database using a core entity constituting the internal spatial knowledge graph as a search term. That is, the knowledge extraction unit (464) can identify an entity named 'Einstein' in the internal SKG and query 'Einstein' in the external SKG database to search for and collect an external knowledge-based knowledge graph such as (Einstein)-[is_a] -> (physicist) and (Einstein)-[born_in] -> (Germany).

[0140] The context integration unit (465) can package both internal and external knowledge and then create a final report, thereby packaging the basic context information, internal SKG, and external SKG into a single Augmented Context and delivering it to the large-scale language model (40). For example, the Augmented Context delivered to the large-scale language model (40) may include a richly augmented and combined context such as, "This person is Einstein (internal SKG), and is sitting on a Paris-style terrace (internal SKG), Einstein is a German-born physicist (external SKG), and Paris is the capital of France (external SKG)."

[0141] Finally, the large-scale language model (40) can receive this augmented context as input and generate a high-level inference answer based on all the information in the context, for example, an answer such as “This person appears to be the physicist Einstein, and the background appears to be influenced by the style of a café terrace in Paris, France.”

[0142] Meanwhile, in this embodiment, the external SKG may be constructed as a Domain-Specific Spatial Knowledge Base specialized in a specific professional field. That is, spatial information of a specific domain, such as medical, architectural, or autonomous driving, may be constructed as a knowledge base, and expert-level spatial information may be searched and transmitted to the model based on the query of the lightweight language model block (460).

[0143] With reference to FIG. 9, a spatial information processing method performed in a spatial information processing system (400) according to a third embodiment of the present invention having the configuration as described above may include: a step of generating a single fusion sequence by combining an image patch and a text token (S410); a step of generating context information between the patch and the token by performing self-attention on the single fusion sequence (S420); a step of generating an internal spatial knowledge graph that structures the current state of an image based on the context information (S430); a step of identifying an entity from the internal SKG and searching for and collecting external knowledge by referring to a pre-built external spatial knowledge graph database using the entity as a query (S440); a step of generating an augmented context by integrating the internal SKG and the collected external knowledge (S450); a step of generating an inference answer by receiving the augmented context (S460); and a step of outputting the inference answer (S470).

[0144] As illustrated in FIG. 5, in the spatial information processing system (500) according to the fourth embodiment of the present invention, the lightweight language model block (560) can suppress hallucinations caused by the inherent spurious pattern of VLM by changing the internal architecture.

[0145] In this embodiment, when image input and query input are input from a user through the input unit (10), the image can be normalized in the spatial transformation network (20), and the vision encoder (30) can convert the image into a sequence of spatial tokens having semantic features. Meanwhile, the query input can be converted into a sequence of language tokens.

[0146] Query input in text form can be processed through natural language processing (NLP). The text of the query input can be passed through a text tokenizer to be broken down into tokens, which are the smallest meaningful units of a sentence. For example, query input text such as "What do you see in this image?" can be converted through the tokenizer into a sequence of language tokens such as [this], [image], [in], [what], [this], [see], [do you], [?]. This sequence of language tokens can be converted into unique vectors (embeddings) and passed to a lightweight language model block (560).

[0147] The vision encoder (30) can understand the input image in semantic units and generate an object or region itself as a single token. That is, the vision encoder (30) can generate time-space complex information indicating 'what' and 'where' is in the image and transmit it to a lightweight language model block (560).

[0148] For semantic tokenization of an image, the vision encoder (30) can scan the entire image using an object detection module (32) or a detailed area caption module (33) and then identify all meaningful elements within the image (object and area detection). For example, elements such as person1 (coordinates), hat (coordinates), snow_slope (area mask), and trees (area mask) can be identified for meaningful objects within the image (spatial information vector).

[0149] Subsequently, the vision encoder (30) can extract a visual feature vector for each detected object / region. This visual feature vector may be a high-dimensional numeric array representing the appearance of the object.

[0150] Finally, the vision encoder (30) can combine the visual feature vector and the spatial information vector of each object / region into one to finally generate a spatial token. Here, the spatial information vector may refer to the bounding box coordinates of the object (e.g., [0.42...]) or a position vector encoded with a region mask. Such a spatial token can be expressed, for example, as tok_person1=[visual feature vector of 'person1']+[position vector of [0.42...] coordinates], tok_hat=[visual feature vector of 'hat']+[position vector of [0.54...] coordinates], tok_snow=[visual feature vector of the 'snow' region]+[position vector of the corresponding region]. The spatial token sequence generated in this way is not a simple pixel piece (patch), but can be a spatial token sequence that already contains rich visual-spatial meaning, such as "a person wearing a blue shirt on the right" or "a red hat at the top center."

[0151] The language token sequence and space token sequence generated in this way are input into the lightweight language model block (560), and attention can proceed in different ways. The lightweight language model block (560) of the fourth embodiment may internally include a bidirectional attention mechanism and a causal attention mechanism. That is, the lightweight language model block (560) of the present embodiment may include a space token processing unit (561) and a language token processing unit (562).

[0152] The spatial token processing unit (561) can identify the spatial layout and interrelationships between spatial tokens of a spatial token sequence received from a vision encoder. The spatial token processing unit (561) can perform bidirectional attention, allowing spatial tokens to freely reference each other, in order to identify the layout and interrelationships of the entire image. For example, by receiving a spatial token sequence such as [tok_snow_slope], [tok_trees], and [tok_sky], and performing bidirectional attention, all tokens within the sequence can freely reference each other without distinguishing between front and back. That is, to determine whether [tok_trees] is above or next to [tok_snow_slope], the two tokens simultaneously reference each other in both directions to understand the structure of the overall scene. In this way, the layout of the entire image is identified at once through bidirectional attention, thereby confirming the spatial context of the image.

[0153] The language token processing unit (562) can generate sequential meanings between language tokens by performing causal attention. That is, causal attention can mask future tokens so that each token can refer only to past tokens that are ahead of it. That is, the language token processing unit (562) applies causal attention to mask future tokens so that they cannot be referenced, as language tokens such as "The", "man", "is..." generated by the lightweight language model are sequential tasks that predict the next word. Through this, when the word "is" is generated, only past tokens such as "The" and "man" can be referenced, so that sentences can be generated sequentially.

[0154] The fourth embodiment of the present invention separates the processing flow of two pieces of information (space and language) at the architectural level, thereby suppressing the hallucinatory phenomenon in which a space token named "snow-covered slope" is habitually (spuriously) combined with a language token named "ski."

[0155] The output of the lightweight language model block (560) is passed to the large-scale language model (40), and the large-scale language model (40) can generate a final answer with suppressed hallucinations based on actual observations.

[0156] With reference to FIG. 10, a spatial information processing method performed in a spatial information processing system (500) according to a fourth embodiment of the present invention having the configuration as described above may include: a step (S510) of generating a spatial token sequence in which a visual feature vector and a spatial information vector are combined from an image; a step (S520) of generating a language token sequence from a text query; a step (S530) of receiving the spatial token sequence and performing bidirectional attention in which tokens refer to each other without distinguishing between front and back to determine the overall spatial layout of the image; a step (S540) of receiving the language token sequence and performing causal attention in which future tokens are masked so that each token refers only to past tokens that are ahead of it to generate sequential language meanings by masking future tokens; a step (S550) of receiving the output of the spatial layout determination operation and the sequential language meaning generation operation to generate a final response; and a step (S560) of outputting the final response.

[0157] Meanwhile, one embodiment of the present invention may be implemented as an Application Specific Integrated Circuit (ASIC) manufactured to suit specific application fields and special functions of devices.

[0158] Custom integrated circuits are also referred to as custom semiconductors. Unlike standard semiconductors, which have fixed specifications and can be applied to any electronic product or application as long as certain requirements are met, custom semiconductors are used for specific products or functions and are integrated circuits manufactured by semiconductor companies to meet specific orders. In other words, custom semiconductors are designed and manufactured to perform only the functions necessary for a specific device or feature. Custom semiconductors are broadly classified according to their design method into Full Custom ICs, which design and manufacture circuits from scratch to meet user requirements, and Semi-Custom ICs, which design and manufacture circuits using parts of a standardized design.

[0159] Application-specific semiconductors are primarily used in communication systems, high-performance computing systems, consumer electronics, automobiles, industrial automation, medical devices, the military, and the aerospace industry; recently, they are being applied to AI semiconductors that execute the large-scale computations required for AI implementation with high performance and power efficiency.

[0160] Application-specific semiconductors (ASICs) are used as core components in communication systems, such as network routers, switches, and modems, performing data packet processing, protocol conversion, and signal processing to provide high throughput and low latency. In high-performance computing systems, ASICs serve as key components for high-speed and parallel processing, while in consumer electronics—including digital cameras, smartphones, tablets, and game consoles—ASICs provide high-performance and low-power solutions required to perform specific functions. In the automotive industry, ASICs are used to control various electronic systems within vehicles, and in industrial automation systems, they provide solutions for high-precision control and high-performance processing.

[0161] An application-specific integrated circuit to which an embodiment of the present invention is applied includes a memory in which an individual memory interface (I / F) is implemented, and may include a plurality of function blocks that request memory access. Each function block may be a Direct Memory Access (DMA) function block, a processor, a video processor, a cache controller, a decompression block, or a data path block. The basic configuration of the application-specific integrated circuit may include a transistor that amplifies or switches an electrical signal, a logic gate which is a circuit that performs a logical function by combining transistors, a memory cell that stores data, an analog circuit which is a circuit that processes continuous voltage or current by combining transistors, and an Intellectual Property Core (IP Core) such as a microprocessor, DSP, or graphics core that is pre-designed to perform a specific function.

[0162] The ASIC may include an individual memory I / F that interfaces with individual memory and an embedded memory I / F that interfaces with embedded memory. The individual memory I / F is connected to each function block to receive memory access signals (e.g., control signals, address signals, and data signals) and, based on these input signals, can generate signals to control the individual memory. The embedded memory I / F is connected to each function block to receive memory access signals (e.g., control signals, address signals, and data signals) and, based on these input signals, can generate modified memory access signals to control the embedded memory. The individual memory I / F and the embedded memory I / F are designed within the memory control block of the ASIC to provide a memory control structure that can be flexibly applied to both the individual memory and the embedded memory.

[0163] Additionally, an application-specific integrated circuit (ASIC) for an artificial neural network (ANN) is composed of multiple neurons arranged in an array and multiple synapse circuits, each neuron being composed of a register, a microprocessor, and at least one input, and each synapse circuit being configured to include memory for storing synapse weights. Here, each neuron of the ASIC may be connected to at least one other neuron through one of the multiple synapse circuits.

[0164] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0165] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0166] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0167] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0168] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0169] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

Claims

1. In a system performed by a computer, At least one memory for storing instructions; and It includes at least one processor that executes the above instructions to perform an operation, The operation performed by executing the instruction by the above at least one processor is, Operation to receive an image and a query; An operation to extract a multidimensional visual feature representation from the above image; The operation of converting the above-mentioned multidimensional visual feature representation into a linguistic representation understandable by a large-scale language model using at least one lightweight artificial intelligence model; and A process including the operation of receiving the above-described converted linguistic expression and the above-described query as input and generating a final response, System.

2. In Claim 1, The above at least one lightweight artificial intelligence model includes a spatial-language mapping neural module (SLMM) separated from the above large-scale language model, and The above-mentioned transforming operation is an operation of transforming numerical coordinate values ​​included in the above-mentioned multidimensional visual feature representation into a spatial semantic vector containing spatial meaning; and The operation of performing cross-attention to map the spatial semantic vector to the corresponding linguistic expression, thereby generating a natural language-based spatial expression report as the linguistic expression; comprising System.

3. In Claim 1, The above-mentioned transforming operation comprises decomposing the image into a plurality of image patches and decomposing the query into a plurality of text tokens and combining them into a single fused sequence; An operation to perform self-attention on the entire single fusion sequence to learn the interrelationships and context between image patches and text tokens; and The operation of generating a structured graph by inferring explicit spatial relationships between objects based on the learned context, and converting the structured graph into text to generate the linguistic expression; System.

4. In Claim 1, The above-mentioned transforming operation is an operation that generates context information by performing self-attention on a single fused sequence combining an image patch and a text token; An operation to generate an internal spatial knowledge graph that structures the current state of an image based on the above context information; The operation of identifying entities from the internal spatial knowledge graph and using the entities as queries to refer to a pre-built external spatial knowledge graph database to search for and collect external knowledge; and The operation of integrating the internal spatial knowledge graph and the collected external knowledge to generate an augmented context, and generating the augmented context into the linguistic expression; System.

5. In Claim 1, The operation of extracting the above-described multidimensional visual feature representation includes the operation of generating a spatial token sequence in which a visual feature vector and a spatial information vector are combined from the image; The above-described transforming operation is an operation of receiving a language token sequence generated from the above query and a spatial token sequence as input, and performing bidirectional attention on the spatial token sequence to determine the overall spatial layout of the image; and An operation to generate sequential language semantics by performing causal attention on the above language token sequence to mask future tokens so that each token refers only to past tokens preceding it; comprising System.

Images