Electronic device and method for searching for image in electronic device

Abstract

An electronic device according to an embodiment comprises: a display; at least one processor; and a memory storing instructions, wherein the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: if inputs of a first image and a first search word for image search are identified, identify, by using a first AI model, a first search vector obtained by converting the first image into a vector, identify, by using a second AI model, a second search vector obtained by converting the first search word into a vector, identify a database (DB) in which a plurality of vectors corresponding to a plurality of objects and a plurality of pieces of image identification information corresponding to a plurality of images are mapped and stored in the memory, identify at least one first candidate vector having a similarity greater than or equal to a first threshold value to the first search vector among the plurality of vectors stored in the DB, identify at least one piece of image identification information mapped to the at least one first candidate vector among the plurality of pieces of image identification information stored in the DB, identify, by using the first AI model, at least one second candidate vector obtained by converting at least one image corresponding to the at least one piece of image identification information into a vector, and display, through the display, at least one image corresponding to the at least one final vector as a search result image, if at least one final vector having a similarity greater than or equal to a second threshold value to the second search vector among the at least one second candidate vector is identified. Other embodiments may be included.

Description

Electronic devices and how to retrieve images from electronic devices

[0001] The present disclosure relates to an electronic device and a method for retrieving an image from an electronic device.

[0002] Multiple images may be stored in an electronic device through shooting or downloading. When the electronic device confirms input of a search image and / or search text by a user, it may display at least one image among the multiple images that corresponds to the search image and / or search text as the search result image.

[0003] An electronic device according to one embodiment may include a display, at least one processor, and a memory for storing instructions. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may, upon confirming the input of a first image and a first search term for image search, use a first AI model to identify a first search vector in which the first image is converted into a vector, and use a second AI model to identify a second search vector in which the first search term is converted into a vector. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may identify a DB in which a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images are mapped and stored in the memory. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one first candidate vector among the plurality of vectors stored in the DB, wherein the similarity to the first search vector is greater than or equal to a first threshold value. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one image identification information among the plurality of image identification information stored in the DB that is mapped to the at least one first candidate vector. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one second candidate vector obtained by converting at least one image corresponding to the at least one image identification information into a vector using the first AI model.When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one final vector among the at least one second candidate vector whose similarity to the second search vector is greater than or equal to a second threshold value, and then display at least one image corresponding to the at least one final vector as a search result image through the display.

[0004] A method for searching for an image in an electronic device according to one embodiment may include, upon confirming the input of a first image and a first search term for image search, verifying a first search vector obtained by converting the first image into a vector using a first AI model, and verifying a second search vector obtained by converting the first search term into a vector using a second AI model. The method according to one embodiment may include an operation of verifying a DB in which a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images are mapped and stored in the memory of the electronic device. The method according to one embodiment may include an operation of verifying at least one first candidate vector among the plurality of vectors stored in the DB of the memory, wherein the similarity to the first search vector is greater than or equal to a first threshold value. The method according to one embodiment may include an operation of verifying at least one image identification information mapped to the at least one first candidate vector among the plurality of image identification information stored in the DB. The method according to one embodiment may include an operation of identifying at least one second candidate vector by converting at least one image corresponding to at least one image identification information into a vector using the first AI model. The method according to one embodiment may include an operation of identifying at least one final vector among the at least one second candidate vector whose similarity to the second search vector is greater than or equal to a second threshold value, and then displaying at least one image corresponding to the at least one final vector as a search result image through the display of the electronic device.

[0005] In a non-volatile storage medium storing instructions according to one embodiment, the instructions are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation may include, upon confirming the input of a first image and a first search term for image search, confirming a first search vector obtained by converting the first image into a vector using a first AI model, and confirming a second search vector obtained by converting the first search term into a vector using a second AI model. The at least one operation according to one embodiment may include an operation to check a DB stored in the memory of the electronic device by mapping a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images. The at least one operation according to one embodiment may include an operation to check at least one first candidate vector among the plurality of vectors stored in the DB of the memory, wherein the similarity to the first search vector is greater than or equal to a first threshold value. According to one embodiment, the at least one operation may include an operation of verifying at least one image identification information mapped to at least one first candidate vector among the plurality of image identification information stored in the DB. According to one embodiment, the at least one operation may include an operation of verifying at least one second candidate vector obtained by converting at least one image corresponding to the at least one image identification information into a vector using the first AI model. According to one embodiment, the at least one operation may include an operation of displaying at least one image corresponding to the at least one final vector as a search result image through the display of the electronic device when at least one final vector among the at least one second candidate vector is verified, wherein the similarity to the second search vector is greater than or equal to a second threshold value.

[0006] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0007] FIG. 2a is a block diagram of an electronic device according to one embodiment, and FIG. 2b is a block diagram for explaining the configuration of a processor and an AI model according to one embodiment.

[0008] FIG. 3 is a diagram illustrating the operation of creating a DB for image search in an electronic device according to one embodiment.

[0009] FIG. 4 is a diagram illustrating the operation of identifying a representative object in an electronic device according to one embodiment.

[0010] FIG. 5 is a diagram illustrating an image search operation in an electronic device according to one embodiment.

[0011] FIGS. 6a, FIGS. 6b, and FIGS. 6c are drawings for illustrating an image search operation in an electronic device according to one embodiment.

[0012] FIG. 7 is a diagram illustrating an image search operation in an electronic device according to one embodiment.

[0013] FIG. 8 is a flowchart illustrating the operation of creating a DB for searching images in an electronic device according to one embodiment.

[0014] FIG. 9 is a flowchart illustrating an operation for searching for an image in an electronic device according to one embodiment.

[0015] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0016] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0017] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0018] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0019] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0020] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0021] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0022] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0023] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0024] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0025] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0026] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0027] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0028] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0029] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0030] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0031] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0032] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0033] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0034] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0035] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0036] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0037] FIG. 2a is a block diagram of an electronic device according to one embodiment, and FIG. 2b is a block diagram for explaining the configuration of a processor and an AI model according to one embodiment.

[0038] Referring to FIGS. 2a and 2b, according to one embodiment, the electronic device (201) may include a processor (220), a memory (230), a display (260), and a communication circuit (290).

[0039] According to one embodiment, the processor (220) can perform overall control operations of the electronic device (201). According to one embodiment, the processor (220) can execute software (e.g., the program (140) of FIG. 1) to control at least one other component (e.g., a hardware or software component) of the electronic device (201) connected to the processor (220), and can perform data processing or operations based on instructions. According to one embodiment, the instructions may include instructions composed of machine language that can be processed by the electronic device (201) or the processor (220). For example, the instructions may include instructions corresponding to operation instructions used in the program.

[0040] According to one embodiment, the processor (220) can generate a plurality of vectors corresponding to a plurality of objects, generate a plurality of image identification information for verifying a plurality of images, and generate a DB (235) (e.g., an image object table) that maps and stores the plurality of vectors and the plurality of identification information, and store it in memory (230).

[0041] According to one embodiment, the processor (220) checks a plurality of images stored in memory (230), detects a plurality of objects (e.g., people, objects and / or animals, etc.) included in each of the plurality of images, and can convert the plurality of objects into a plurality of vectors by embedding them using a first AI model (231a).

[0042] According to one embodiment, the first AI model (231a) may include an image encoder.

[0043] According to one embodiment, the processor (220) can use an image object detection unit (221) to check a plurality of images stored in memory (230) and detect a plurality of objects (e.g., people, objects and / or animals, etc.) included in each of the plurality of images.

[0044] According to one embodiment, the processor (220) may include an image object detection unit (221) or control a separately configured image object detection unit (221).

[0045] According to one embodiment, the processor (220) can perform the same operation as the image object detection unit (221).

[0046] According to one embodiment, the processor (220) can generate multiple image identification information for verifying multiple images.

[0047] According to one embodiment, when creating a DB (235) for image search, the processor (220) assigns a unique number to each of the multiple images that can identify the multiple images, and can identify the assigned unique number as an image identification number.

[0048] According to one embodiment, the processor (220) can identify multiple image identification information corresponding to (assigned to) multiple images containing each of multiple objects corresponding to multiple vectors, and map multiple vectors to multiple image identification information to create a DB (235) and store it in memory (230).

[0049] According to one embodiment, the processor (220) can store at least one image identification number mapped to one vector corresponding to one object in the DB (235).

[0050] According to one embodiment, the processor (220) can identify a plurality of first objects in which the number of identical objects among the plurality of objects is greater than or equal to a specified number, identify the plurality of first objects as important objects, and identify one representative object among the plurality of first objects.

[0051] According to one embodiment, the processor (220) can identify a plurality of first objects in which the number of identical objects among a plurality of objects is greater than a specified number, create a cluster for the plurality of first objects, and use a cluster algorithm (e.g., K-means and / or DB-Scan) to select a central object among the plurality of first objects and identify the central object as a representative object.

[0052] According to one embodiment, the processor (220) can use the first AI model (231a) to embedding a representative object and converting it into a vector, identify multiple image identification information corresponding to multiple images including the representative object, and map and store the vector corresponding to the representative object and the multiple image identification information in the DB (235).

[0053] According to one embodiment, if the processor (220) identifies at least one second object among a plurality of objects in which the number of identical objects is less than or equal to a specified number, it identifies at least one second object as an unimportant object (e.g., an individual object that appears fragmentarily) and may not use at least one second object in the DB (235) (e.g., an image object table).

[0054] According to one embodiment, when the processor (220) checks the input of a first image and a first search term for image search, it can check a first search vector converted into a vector by embedding the first image using a first AI model (231a), and check a second search vector converted into a vector by embedding the first search term using a second AI model (231b).

[0055] According to one embodiment, the second AI model (231b) may include a text encoder.

[0056] According to one embodiment, the processor (220) can check a DB (235) (e.g., an image object table) that stores a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images in memory (230).

[0057] According to one embodiment, the processor (220) compares the similarity between a plurality of vectors stored in DB (235) and a first search vector converted from a first image for image search, and can identify at least one first candidate vector among the plurality of vectors whose similarity to the first search vector is greater than or equal to a first threshold value.

[0058] According to one embodiment, when the processor (220) checks at least one first candidate vector, it can check at least one image identification information mapped to at least one first candidate vector among a plurality of image identification information stored in DB (235).

[0059] According to one embodiment, when the processor (220) checks at least one image identification information mapped to at least one first candidate vector, it can check at least one image among a plurality of images stored in memory (230) that corresponds to at least one image identification information mapped to at least one first candidate vector (at least one identification information assigned thereto).

[0060] According to one embodiment, when the processor (220) identifies at least one image corresponding to at least one image identification information, it can identify at least one second candidate vector by using the first AI model (231a) to embedding each of the at least one image corresponding to at least one image identification information and converting it into a vector.

[0061] According to one embodiment, the processor (220) compares the similarity between at least one second candidate vector and a second search vector that has converted a first search term for image search into a vector, and can identify at least one final vector among the at least one second candidate vector that has a similarity with the second search vector greater than or equal to a second threshold value.

[0062] According to one embodiment, the processor (220) can identify at least one image corresponding to at least one final vector among at least one image corresponding to at least one image identification information when identifying at least one final vector.

[0063] According to one embodiment, the processor (220) can display at least one image corresponding to at least one final vector as a search result image through the display (260).

[0064] According to one embodiment, the processor (220), upon confirming input of a first image and a first search term for image search, detects at least one object included in the first image, and using a first AI model (231a), embedding each of the at least one object detected in the first image to convert it into at least one object, and can use a vector corresponding to the at least one object detected in the first image as a first search vector.

[0065] According to one embodiment, when the processor (220) checks the input of a first image for image search, it can identify at least one keyword designated for image search based on the frequency of use of search terms (keywords) by the user of the electronic device (201).

[0066] According to one embodiment, when the processor (220) checks the input of a first image for image search, it can identify at least one keyword used more than a specified number of times among the search terms (keywords) used by the user of the electronic device (201).

[0067] According to one embodiment, the processor (220) can identify at least one keyword that has been pre-analyzed by selecting a specified number based on the average search statistics service of device users collected as big data, and provide the identified at least one keyword to the user of the electronic device (201).

[0068] According to one embodiment, the processor (220) can identify a first search vector converted into a vector by embedding a first image using a first AI model (231a), and identify a vector corresponding to at least one keyword converted into a vector by embedding at least one keyword using a second AI model (231b).

[0069] According to one embodiment, the processor (220) can identify at least one image identification information that maps to at least one first candidate vector whose similarity to a first search vector among a plurality of image identification information stored in DB (235) is confirmed to be greater than or equal to a first threshold value, and can identify at least one second candidate vector that converts at least one image corresponding to at least one image identification information into a vector.

[0070] According to one embodiment, when the processor (220) identifies at least one final vector having a similarity to at least one keyword among at least one second candidate vector that is greater than or equal to a second threshold value, it can identify at least one image corresponding to at least one final vector among at least one image corresponding to at least one image identification information as a search result image.

[0071] According to one embodiment, the processor (220) can distinguish search result images by at least one keyword and display the search result images distinguished by at least one keyword through the display (260).

[0072] According to one embodiment, the processor (220) can generate a vector corresponding to the first keyword and a vector corresponding to the second keyword using a second AI model (231b) when at least one keyword includes the first keyword and the second keyword.

[0073] According to one embodiment, when the processor (220) identifies at least one first final vector having a similarity to a vector corresponding to a first keyword among at least one second candidate vector that is greater than or equal to a second threshold value, it can identify at least one image corresponding to at least one first final vector as a first search result image.

[0074] According to one embodiment, when the processor (220) identifies at least one second final vector whose similarity to a vector corresponding to a second keyword among at least one second candidate vector is greater than or equal to a second threshold value, it can identify at least one image corresponding to at least one second final vector as a second search result image.

[0075] According to one embodiment, the processor (220) can distinguish and display a first search result image corresponding to a first keyword and a second search result image corresponding to a second keyword through a display (260).

[0076] A processor (220) according to one embodiment can generate a personalized album in which search result images corresponding to one object are classified by at least one keyword.

[0077] A processor (220) according to one embodiment can identify a plurality of first objects in which the number of identical objects among a plurality of objects detected from a plurality of images stored in memory (230) is greater than or equal to a specified number, identify a representative object corresponding to a central object among the plurality of first objects, and convert the representative object into a vector using a first AI model (231a).

[0078] A processor (220) according to one embodiment can identify at least one keyword for image search based on the frequency of use of a user's search term (keyword) of an electronic device (201), and can identify a vector corresponding to at least one keyword by embedding at least one keyword and converting it into a vector using a second AI model (231b).

[0079] According to one embodiment, the processor (220) can identify at least one image identification information that maps to at least one first candidate vector, which is confirmed to have a similarity degree to a vector corresponding to a representative object among a plurality of image identification information stored in DB (235) that is greater than or equal to a first threshold value, and can identify at least one second candidate vector that converts at least one image corresponding to at least one image identification information into a vector.

[0080] According to one embodiment, when the processor (220) identifies at least one final vector having a similarity to at least one keyword among at least one second candidate vector that is greater than or equal to a second threshold value, it can identify at least one image corresponding to at least one final vector as a search result image.

[0081] According to one embodiment, the processor (220) can generate a personalized album that distinguishes search result images corresponding to representative objects for at least one keyword, and can display the personalized album through the display (260).

[0082] According to one embodiment, when the processor (220) confirms the input of a first image for image search, it can recommend at least one keyword mapped to a plurality of object and plurality of image identification information in the DB (235) as a search term.

[0083] According to one embodiment, the processor (220) can detect at least one keyword from the plurality of images by identifying the plurality of images corresponding to the plurality of image identification information stored in the DB (235) from the memory (230) and interpreting each of the plurality of images using the third AI model (231c).

[0084] According to one embodiment, the third AI model (231c) includes an image descriptor model and may include a large language model (LLM).

[0085] According to one embodiment, the processor (220) may store at least one keyword corresponding to each of the multiple vectors and multiple image identification information by additionally mapping it to a DB (235) (e.g., image object table) that stores the mapping of multiple vectors and multiple image identification information.

[0086] According to one embodiment, when the processor (220) checks the input of a first image for image search, it checks a first search vector converted into a vector by embedding the first image using a first AI model (231a), and checks at least one first candidate vector among a plurality of vectors stored in the DB (235) that has a similarity to the first search vector greater than or equal to a first threshold value.

[0087] According to one embodiment, when the processor (220) identifies at least one keyword mapped to at least one first candidate vector in the DB (235), it can display at least one keyword mapped to at least one first candidate vector as a recommended search term through the display (260).

[0088] According to one embodiment, when the processor (220) confirms the selection of a third keyword among the at least one keywords while displaying at least one keyword as a recommended search term through the display (260), it can check at least one identification information mapped to the third keyword in the DB (235) and check at least one image corresponding to the at least one identification information as a search result image.

[0089] According to one embodiment, the processor (220) can display a first image and a search result image searched with a third keyword through a display (260).

[0090] According to one embodiment, when displaying search result images through a display (260), the processor (220) can determine a priority order for displaying search result images in order of highest similarity based on similarity with a first search vector and / or a second search vector.

[0091] According to one embodiment, the processor (220) may display at least one first search result image (e.g., 3) in a specified number in order of having the highest similarity among the search result images in a first area of ​​the display (260).

[0092] According to one embodiment, the processor (220) may display at least one second search result image, excluding at least one first search result image among the search result images, as a list in a second area located below the first area of ​​the display (260) in order of highest similarity.

[0093] According to one embodiment, when displaying search result images through a display (260), the processor (220) checks the score using metadata of the search result images and can determine the priority order for displaying search result images in order of having the highest score based on the score.

[0094] According to one embodiment, the processor (220) can determine the number of clicks on an image, the number of shares on an image, and / or the number of searches on an image from the metadata of the search result image, and calculate a score based on the following <Equation 1>. The following <Equation 1> is merely an example for calculating a score, and the score can be calculated in various ways.

[0095] <Equation 1>

[0096] Score = αA + βB + γC

[0097] α: Number of image click weights

[0098] β: Number of image share weights

[0099] γ: Number of image search weights

[0100] According to one embodiment, the processor (220) may display at least one first search result image in a specified number (e.g., 3) in order of having the highest score among the search result images in a first area of ​​the display (260).

[0101] According to one embodiment, the processor (220) may display at least one second search result image, excluding at least one first search result image among the search result images, as a list in a second area located below the first area of ​​the display (260) in order of having the highest score.

[0102] According to one embodiment, the processor (220) may display at least one first search result image of a specified number (e.g., 3) in a first area of ​​the display (260) that has the highest order of similarity to the first search vector and / or the second search vector among the search result images.

[0103] According to one embodiment, the processor (220) can check the score using metadata of at least one second search result image excluding at least one first search result image among the search result images, and based on the score, display at least one second search result image as a list in a second area located below the first area of ​​the electronic device's display in order of highest score.

[0104] According to one embodiment, the processor (220) can determine the priority order for displaying search result images by using conditions such as the number of saves, the number of shares, whether uploaded to a server, the number of searches, the number of edits, whether downloaded, and / or whether a tag such as "like" is assigned to an image included in a category such as a first image and / or a first search term for image search.

[0105] According to one embodiment, the processor (220) can determine the priority order for displaying search result images by using conditions such as whether the first image and / or first search term for image search belong to the same category, user usage history information and / or user preference.

[0106] A memory (230) according to one embodiment may be implemented substantially identically or similarly to the memory (130) of FIG. 1.

[0107] In one embodiment, the memory (230) may store a DB (235) including an image object table.

[0108] According to one embodiment, an on-device AI model (231) may be stored in the memory (230), and the on-device AI model (231) may be configured in various ways, such as including a first AI model (231a), a second AI model (231b), and a third AI model (231c), or including at least one AI model among the first AI model (231a), the second AI model (231b), and the third AI model (231c), or including at least one of the first AI model (231a), the second AI model (231b), and the third AI model (231c) as one identical AI model, or including the first AI model (231a), the second AI model (231b), and the third AI model (231c) as one identical AI model.

[0109] At least some of the on-device AI models (231) according to one embodiment can be operated as external AI models.

[0110] According to one embodiment, the on-device AI model (231) performs processing of sensitive information such as personal information, and the external AI model can perform processing of information excluding sensitive information such as personal information.

[0111] An on-device AI model (231) according to one embodiment is an artificial intelligence model implemented in an electronic device (201) and can provide various functions without a network.

[0112] According to one embodiment, a plurality of artificial intelligence models may be stored in the memory (230).

[0113] According to one embodiment, each of the plurality of AI models may be a model trained based on a specified type of learning algorithm, and may be an AI model implemented to receive various types of data (or content) as input, perform calculations, and output (or obtain) result data.

[0114] According to one embodiment, a plurality of AI models may include generative AI models.

[0115] A generative AI model according to one embodiment can generate and output new content (e.g., text, images, and / or computer code, etc.) based on learned content in response to an input prompt. For example, in an electronic device (601), learning is performed to output specific types of result data as output data using data of specified types based on a machine learning algorithm or a deep learning algorithm, thereby generating multiple AI models (e.g., machine learning models and deep learning models) that are stored in the electronic device (601), or AI models learned from an external electronic device (e.g., an external server) can be transmitted to and stored in the electronic device (601). For example, the electronic device (601) can output input data (input data) as output data of a model learned through artificial intelligence of specified types based on a machine learning algorithm or a deep learning algorithm. Machine learning algorithms include supervised learning algorithms such as linear regression and logistic regression, unsupervised learning algorithms such as clustering, visualization and dimensionality reduction, and association rule learning, and reinforcement learning algorithms, and deep learning algorithms may include Artificial Neural Networks (ANN), Deep Neural Networks (DNN), and Convolutional Neural Networks (CNN), and may further include various learning algorithms not limited to those described.The trained AI model includes at least one operation (e.g., a convolution layer or a pooling layer) for processing input data, and can be implemented to output result data by performing operations on the input data based on at least one operation.

[0116] According to one embodiment, an application that can be connected to an external AI model (281) may be stored in the memory (230).

[0117] According to one embodiment, the external AI model (281) may include each of the first AI model (231a), the second AI model (231b), and / or the third AI model (231c), or include at least one of the first AI model (231a), the second AI model (231b), and / or the third AI model (231c), or include the first AI model (231a), the second AI model (231b), and / or the third AI model (231c) as one identical AI model.

[0118] A display (260) according to one embodiment may be implemented substantially identically or similarly to the display (160) of FIG. 1.

[0119] A display (260) according to one embodiment may display a search result image based on the input of a first image and / or a first search term for image search.

[0120] A communication circuit (290) according to one embodiment can form a communication connection with an external electronic device (e.g., another electronic device, or a server) using various types of communication methods and can transmit and / or receive data. As described above, the communication methods may include a communication method that establishes a direct communication connection such as Bluetooth and / or Wi-Fi Direct, a communication method that uses an access point (AP) (e.g., Wi-Fi communication), or a communication method that uses cellular communication using a base station (e.g., 3G, 4G / LTE, 5G). Since the communication circuit (290) can be implemented as described above in the communication module (190) in FIG. 1, a redundant description is omitted.

[0121] FIG. 3 is a diagram illustrating the operation of creating a DB for image search in an electronic device according to one embodiment.

[0122] Referring to FIG. 3, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can check a plurality of images (311) stored in the memory of the electronic device (e.g., the memory (230) of FIG. 2a).

[0123] According to one embodiment, the electronic device can detect a plurality of objects (313) included in each of a plurality of images (311).

[0124] According to one embodiment, the electronic device can use an image object detection unit of the electronic device (e.g., image object detection unit (221) of FIG. 2A) to check a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2A) and detect a plurality of objects (e.g., people, objects and / or animals, etc.) included in each of the plurality of images.

[0125] According to one embodiment, the electronic device can convert a plurality of objects (313) into a plurality of vectors (315) by embedding them using a first AI model (231a) (e.g., the first AI model (231a) of FIG. 2B).

[0126] According to one embodiment, the electronic device assigns a unique number to each of the plurality of images (311) that can identify the plurality of images, and can identify the unique number as an image identification number (e.g., "image id: k", "image id: k+1", ...) (317).

[0127] According to one embodiment, the electronic device can identify a plurality of image identification information (317) corresponding to a plurality of images (311) each containing a plurality of objects (313) corresponding to a plurality of vectors (315), map the plurality of vectors (315) to the plurality of image identification information (317) to create a DB (235) (e.g., the DB (235) of FIG. 2a) (image object mapping table) and store it in the memory of the electronic device (memory (230) of FIG. 2a).

[0128] FIG. 4 is a diagram illustrating the operation of identifying a representative object in an electronic device according to one embodiment.

[0129] Referring to FIG. 4, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can check a plurality of images stored in the memory of the electronic device (e.g., the memory (230) of FIG. 2a) and detect a plurality of objects included in each of the plurality of images.

[0130] According to one embodiment, the electronic device can generate a cluster (413) for a plurality of first objects (411) when it identifies a plurality of first objects (411) in which the same object among a plurality of objects detected in each of a plurality of images is greater than or equal to a specified number.

[0131] According to one embodiment, the electronic device can select a central object from a cluster (413) of a plurality of first objects using a cluster algorithm (e.g., K-means and / or DB-Scan) and identify the central object as a representative object (415).

[0132] According to one embodiment, the electronic device can use a first AI model (e.g., the first AI model (231a) of FIG. 2b) to embedding a representative object (415) and converting it into a vector, identify a plurality of image identification information (e.g., "image id: n", "image id: n+1", "image id: n+2") corresponding to a plurality of images including the representative object (415), and store a DB (235) in the memory of the electronic device (e.g., memory (230) of FIG. 2a) by mapping a vector (417) corresponding to the representative object with the plurality of image identification information (419).

[0133] FIG. 5 is a diagram illustrating an image search operation in an electronic device according to one embodiment.

[0134] Referring to FIG. 5, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can receive input of a first image (501) and a first search term (503) for image search.

[0135] According to one embodiment, the electronic device can verify a first search vector converted into a vector by embedding a first image (501) using a first AI model (231a) (e.g., the first AI model (231a) of FIG. 2b).

[0136] According to one embodiment, the electronic device can verify a second search vector converted into a vector by embedding the first search term (503) using a second AI model (e.g., the second AI model (231b) of FIG. 2b).

[0137] According to one embodiment, the electronic device can check a DB (235) (e.g., image object table) that stores a plurality of vectors (505) corresponding to a plurality of objects and a plurality of image identification information (507) corresponding to a plurality of images by mapping them in the memory of the electronic device (e.g., memory (230) of FIG. 2a).

[0138] According to one embodiment, the electronic device can compare the similarity between a plurality of vectors (505) stored in a DB (235) (e.g., DB (235) of FIG. 2a) and a first search vector converted from a first image for image search (509).

[0139] According to one embodiment, the electronic device can identify at least one first candidate vector (511) among a plurality of vectors (505) such that the similarity with the first search vector is greater than or equal to a first threshold value (α).

[0140] According to one embodiment, when the electronic device identifies at least one first candidate vector (511), it can identify at least one image identification information (513) mapped to at least one first candidate vector (511) among a plurality of image identification information (507) stored in the DB (235).

[0141] According to one embodiment, when an electronic device checks at least one image identification information (513), it can identify at least one image corresponding to at least one image identification information (513) among a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a).

[0142] According to one embodiment, the electronic device can identify at least one second candidate vector (515) by embedding each of at least one image corresponding to at least one image identification information (513) using a first AI model (231a) (e.g., the first AI model (231a) of FIG. 2b) and converting it into a vector.

[0143] According to one embodiment, the electronic device can compare the similarity between at least one second candidate vector (515) and a second search vector that converts a first search term for image search into a vector (517).

[0144] According to one embodiment, the electronic device can identify at least one final vector among at least one second candidate vector (515) whose similarity to the second search vector is greater than or equal to a second threshold value (β).

[0145] According to one embodiment, when the electronic device identifies at least one final vector, it can identify at least one image identification information (519) corresponding to at least one final vector among at least one image identification information (513).

[0146] According to one embodiment, the electronic device may display at least one image corresponding to at least one image identification information (519) as a search result image (521) through the display of the electronic device (e.g., the display (260) of FIG. 2a).

[0147] FIGS. 6a, FIGS. 6b, and FIGS. 6c are drawings for illustrating an image search operation in an electronic device according to one embodiment.

[0148] Referring to FIG. 6a, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can identify at least one keyword (603) (e.g., Camping, Eat out, Wedding ceremony) that has been used more than a specified number of times among the search terms (keywords) used by the user of the electronic device when checking the input of a first image (601) for image search.

[0149] According to one embodiment, the electronic device can identify a first search vector converted into a vector by embedding a first image (601) using a first AI model (e.g., the first AI model (231a) of FIG. 2b), and identify a vector (605) corresponding to at least one keyword by embedding at least one keyword (603) using a second AI model (231b) (e.g., the second AI model (231b) of FIG. 2b).

[0150] According to one embodiment, the electronic device can identify at least one first candidate vector (611) (e.g., Img Emb #2) among a plurality of vectors (607) stored in the electronic device's DB (235) (e.g., DB (235) of FIG. 2a) that has a similarity to a first search vector greater than or equal to a first threshold value.

[0151] According to one embodiment, the electronic device can identify at least one image identification information (613) (e.g., k, k+1, k+2, k+3, k+4, k+5,..) that maps to at least one first candidate vector (611) among a plurality of image identification information (609) stored in DB (235).

[0152] According to one embodiment, the electronic device can identify at least one second candidate vector by converting at least one image corresponding to at least one image identification information (613) into a vector using a first AI model (e.g., the first AI model (231a) of FIG. 2a).

[0153] According to one embodiment, the electronic device can compare the similarity between at least one second candidate vector and at least one vector (605) corresponding to a keyword (615).

[0154] According to one embodiment, when an electronic device identifies at least one final vector having a similarity to at least one keyword among at least one second candidate vector (605) that is greater than or equal to a second threshold value (β), it can identify at least one image identification information (617) corresponding to at least one final vector among at least one image identification information (613).

[0155] According to one embodiment, the electronic device may divide at least one image identification information (617) corresponding to at least one final vector into at least one first image identification information (617a) corresponding to a first keyword (e.g., Camping) among at least one keyword, and at least one second image identification information (617b) corresponding to a second keyword (e.g., Eat out) among at least one keyword.

[0156] According to one embodiment, the electronic device can identify at least one image corresponding to at least one image identification information (617) among at least one image corresponding to at least one image identification information (613) as a search result image.

[0157] According to one embodiment, the electronic device can distinguish search result images by at least one keyword and display the search result images distinguished by at least one keyword through the display of the electronic device (e.g., the display (260) of FIG. 2a).

[0158] Referring to FIG. 6b for a detailed description, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can, upon confirming the input of a first image (631) for image search, use a first AI model (e.g., the first AI model (231a) of FIG. 2b) to confirm a first search vector in which the first image (631) is embedded and converted into a vector (operation 633).

[0159] According to one embodiment, the electronic device can compare the similarity between a plurality of vectors stored in the electronic device's DB (e.g., DB (235) of FIG. 2a) and a first search vector (operation 635).

[0160] According to one embodiment, the electronic device performs an operation (637) to identify a vector among a plurality of vectors stored in the electronic device's DB (e.g., DB (235) of FIG. 2a) that has a similarity to a first search vector greater than or equal to a first threshold value (α), and if the similarity is less than the first threshold value, the image search operation may be terminated (639).

[0161] According to one embodiment, the electronic device identifies at least one vector among a plurality of vectors stored in the electronic device's DB (e.g., DB (235) of FIG. 2a) that has a similarity to a first search vector greater than or equal to a first threshold value (α) (operation 637), and if the similarity is greater than the first threshold value as a result of the identification, at least one vector among the plurality of vectors that has a similarity to a first search vector greater than or equal to the first threshold value (α) can be identified as at least one first candidate vector (operation 641).

[0162] According to one embodiment, the electronic device can identify at least one image identification information that maps to at least one first candidate vector among a plurality of image identification information stored in the electronic device's DB (e.g., DB (235) of FIG. 2a), and identify at least one second candidate vector that converts at least one image corresponding to at least one image identification information into a vector (operation 643).

[0163] According to one embodiment, the electronic device can identify at least one specified keyword based on the frequency of use of the user's search term (keyword) (operation 645).

[0164] According to one embodiment, the electronic device can identify a first keyword (e.g., Camping) (k1) and a second keyword (e.g., Eat out) (k2) included in at least one specified keyword (operation 647).

[0165] According to one embodiment, the electronic device can identify a vector (649a) corresponding to a first keyword (e.g., Camping) converted into a vector by embedding a first keyword (e.g., Camping) and a vector (649b) corresponding to a second keyword (e.g., Eat out) converted into a vector by embedding a second keyword (e.g., Eat out) using a second AI model (e.g., the second AI model (231b) of FIG. 2b).

[0166] According to one embodiment, the electronic device can compare the similarity between at least one second candidate vector and a vector (649a) corresponding to the first keyword (operation 651).

[0167] According to one embodiment, the electronic device identifies a vector among at least one second candidate vector that has a similarity to a vector (649a) corresponding to a first keyword and has a similarity greater than or equal to a second threshold value (operation 653), and if the similarity is not greater than the second threshold value, the image search operation may be terminated (operation 655).

[0168] According to one embodiment, the electronic device identifies at least one final vector (653 operation) that has a similarity to at least one vector (649a) corresponding to a first keyword among at least one second candidate vector, and if the similarity is at least one image identification information (657) corresponding to at least one final vector among a plurality of image identification information stored in the DB of the electronic device, and can classify at least one image identification information (657) corresponding to the first keyword (e.g., Camping).

[0169] According to one embodiment, the electronic device can compare the similarity between at least one second candidate vector and a vector (649b) corresponding to the second keyword (operation 651).

[0170] According to one embodiment, the electronic device identifies a vector that has a similarity to a second keyword (649b) among at least one second candidate vector and is greater than or equal to a second threshold value (operation 653), and if the similarity is less than the second threshold value, the image search operation may be terminated (operation 655).

[0171] According to one embodiment, the electronic device identifies at least one final vector (653 operation) that has a similarity to at least one second keyword among at least one second candidate vector (649b) and is greater than or equal to a second threshold value, and if the similarity is greater than or equal to the second threshold value, identifies at least one image identification information (659) that corresponds to at least one final vector among a plurality of image identification information stored in the DB of the electronic device, and can classify at least one image identification information (659) as at least one image identification information (659) that corresponds to the second keyword (e.g., Eat out).

[0172] Referring to FIG. 6c, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can identify at least one image identification information for each specified at least one keyword through an operation such as FIG. 6a and FIG. 6b, and identify at least one image (671, 673 and 675) corresponding to at least one image identification information among a plurality of images stored in the memory of the electronic device.

[0173] According to one embodiment, the electronic device may distinguish and display at least one image (671) corresponding to a first keyword (k1), for example, an image (671) corresponding to camping when the first keyword is camping, at least one image (673) corresponding to a second keyword (k2), for example, an image (673) corresponding to eating out when the second keyword is eating out, and / or at least one image (675) corresponding to a third keyword (k3), for example, an image (675) corresponding to a third keyword (k3).

[0174] FIG. 7 is a diagram illustrating an image search operation in an electronic device according to one embodiment.

[0175] Referring to FIG. 7, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can generate a DB (235) (e.g., the DB (235) of FIG. 2a) that maps a plurality of vectors (711) corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images.

[0176] According to one embodiment, the electronic device can detect at least one keyword (701) (e.g., Camping, Wedding, and / or Eat out) from the plurality of images by identifying the plurality of images corresponding to the plurality of image identification information (713) stored in the DB (235) of the electronic device among the plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a), and interpreting each of the plurality of images using a third AI model (231c) (e.g., third AI model (231c) of FIG. 2c).

[0177] According to one embodiment, the electronic device may store at least one keyword (715) corresponding to each of the plurality of vectors and the plurality of image identification information (713) by additionally mapping them to a DB (235) (e.g., an image object table) that stores a plurality of vectors (711) and a plurality of image identification information (713).

[0178] According to one embodiment, when the electronic device checks the input of a first image (703) for image search, it checks a first search vector converted into a vector by embedding the first image (703) using a first AI model (e.g., the first AI model (231a) of FIG. 2a), and checks at least one first candidate vector (705) among a plurality of vectors stored in DB (235) that has a similarity to the first search vector greater than or equal to a first threshold value.

[0179] According to one embodiment, when an electronic device identifies at least one keyword (707) (e.g., Wedding, Camping, and Eat out) mapped to at least one first candidate vector (705) in DB (235), it can display at least one keyword (707) mapped to at least one first candidate vector (705) as a recommended search term (709) through the display of the electronic device (e.g., the display (260) of FIG. 2a).

[0180] According to one embodiment, the electronic device may provide at least one keyword to be input into a second AI model (e.g., a second AI model (231b) of FIG. 2b) (a text encoder) instead of a specified at least one keyword as a category-specific automatically generated and / or recommended search term, depending on the type of first image for image search (e.g., a pre-trained and modeled image) input into a first AI model (e.g., a first AI model (231a) of FIG. 2b) (an image encoder).

[0181] According to one embodiment, the electronic device can automatically recommend and / or select an image selected by a user of the electronic device or an image based on specified conditions as a first image for image search.

[0182] According to one embodiment, the electronic device may use at least one specified keyword (e.g., camping, eat out, and / or wedding ceremony) as a text prompt to retrieve at least one image in the memory of the electronic device (e.g., memory (230) of FIG. 2a).

[0183] According to one embodiment, the electronic device can learn in advance a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a) using an AI model (e.g., 231 of FIG. 2b) based on a first image for image search selected by a user of the electronic device among a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a) for image search, so as to be able to automatically generate at least one keyword input into the second AI model (231b) (text encoder) of FIG. 2b by category).

[0184] According to one embodiment, when the electronic device confirms the selection of a first image for image search by a user of the electronic device among a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a) during the course of time, the electronic device can generate a category based on a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a) using an AI model (e.g., 231 of FIG. 2b)).

[0185] According to one embodiment, the electronic device may provide at least one keyword as a recommended search term, which is generated by pre-learning a first image for image search among a plurality of images stored in memory (e.g., memory (230) of FIG. 2a), rather than at least one designated keyword. An electronic device according to one embodiment (101 of FIG. 1; 201 of FIG. 2a to 2b)) may include a display (160 of FIG. 1; 260 of FIG. 2a), at least one processor (120 of FIG. 1; 220 of FIG. 2a); and a memory for storing instructions (130 of FIG. 1; 230 of FIG. 2a). When the above commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can, upon confirming the input of a first image and a first search term for image search, use a first AI model (231a in FIG. 2b) to identify a first search vector in which the first image is converted into a vector, and use a second AI model (231b in FIG. 2b) to identify a second search vector in which the first search term is converted into a vector. When the above commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify a DB in which a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images are mapped and stored in the memory. When the above commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one first candidate vector among the plurality of vectors stored in the DB, wherein the similarity to the first search vector is greater than or equal to a first threshold value.When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one image identification information mapped to at least one first candidate vector among the plurality of image identification information stored in the DB. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one second candidate vector by converting at least one image corresponding to the at least one image identification information into a vector using the first AI model. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one final vector among the at least one second candidate vector whose similarity to the second search vector is greater than or equal to a second threshold value, and then display at least one image corresponding to the at least one final vector as a search result image through the display.

[0186] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can detect the plurality of objects in each of the plurality of images stored in the memory. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can convert the plurality of objects into a plurality of vectors using the first AI. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can verify the plurality of identification information corresponding to the plurality of images containing each of the plurality of objects. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can map the plurality of vectors and the plurality of image identification information and store them in the DB.

[0187] When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify a plurality of first objects in which the same object among the plurality of objects is greater than or equal to a specified number, and can identify a representative object corresponding to a central object among the plurality of first objects using a cluster algorithm. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can convert the representative object into a vector using the first AI. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify a plurality of image identification information corresponding to a plurality of images including the representative object. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can map the vector corresponding to the representative object with the plurality of image identification information and store it in the DB.

[0188] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may not use the at least one second object in the DB if it identifies at least one second object among the plurality of objects in which the number of identical objects is less than or equal to the specified number.

[0189] When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one keyword based on the frequency of search term usage by the user of the electronic device when confirming the input of the first image for image search. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can convert the at least one keyword into a vector using the second AI model. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one final vector among the at least one second candidate vectors whose similarity to the vector corresponding to the at least one keyword is greater than or equal to the second threshold value, and then display at least one image corresponding to the at least one final vector as a search result image through the display.

[0190] When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can detect at least one keyword from a plurality of images corresponding to the plurality of image identification information using a third AI model. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can map and store the detected at least one keyword to the plurality of vectors and the plurality of image identification information in the DB. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify a first search vector obtained by converting the first image into a vector using the first AI model when verifying the input of the first image for image search. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one first candidate vector among the plurality of vectors stored in the DB, wherein the similarity to the first search vector is greater than or equal to a first threshold value. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify at least one keyword mapped to the at least one first candidate vector in the DB. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can recommend the at least one keyword as a search term for the image search.

[0191] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may display at least one first search result image of a specified number having the highest similarity to the first search vector and / or the second search vector among the search result images in a first area among the display areas. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may display at least one second search result image of the search result images, excluding the at least one first search result image, in a list in a second area located below the first area among the display areas, in the order of highest similarity to the first search vector and / or the second search vector.

[0192] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may check a score using metadata of the search result image in a first area among the display areas, and display at least one first search result image of a specified number having the highest score among the search result images based on the score. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may check a score using metadata of at least one second search result image excluding the at least one first search result image among the search result images in a second area located below the first area among the display areas, and display at least one second search result image in a list in order of highest score based on the score.

[0193] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may display at least one first search result image of a specified number having the highest similarity to the first search vector and / or the second search vector among the search result images in a first area among the display areas. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may check a score using the metadata of at least one second search result image excluding the at least one first search result image among the search result images in a second area located below the first area among the display areas, and display the at least one second search result image in order of highest score based on the score as a list containing the at least one second search result image.

[0194] FIG. 8 is a flowchart illustrating the operation of creating a database for searching images in an electronic device according to one embodiment. The operations for creating a database for searching images may include operations 801 through 809. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0195] In operation 801, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can detect multiple objects included in each of the multiple images.

[0196] According to one embodiment, the electronic device can check a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a) and detect a plurality of objects (e.g., people, objects and / or animals, etc.) included in each of the plurality of images.

[0197] According to one embodiment, the electronic device can extract an object using an image object extractor (e.g., the image object extractor (221) of FIG. 2a). The image object extractor may include, for example, artificial intelligence, and may include artificial intelligence that distinguishes objects through training from images and analyzes images capable of recognizing objects.

[0198] According to one embodiment, the image object detection unit can extract objects based on various multimedia data, such as images and / or videos.

[0199] In operation 803, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can embed a plurality of objects and convert them into a plurality of vectors.

[0200] According to one embodiment, the electronic device can convert a plurality of objects into a plurality of vectors by embedding them using a first AI model (e.g., the first AI model (231a) of FIG. 2b).

[0201] In operation 805, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can generate multiple image identification information for verifying multiple images.

[0202] According to one embodiment, when an electronic device creates a DB for image search (e.g., DB (235) of FIG. 2a) (e.g., image object mapping table), it assigns a unique number to each of the multiple images that can identify the multiple images, and can identify the unique number as an image identification number.

[0203] In operation 807, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can identify multiple image identification information corresponding to multiple images containing each of multiple objects corresponding to multiple vectors.

[0204] In operation 809, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can generate a DB (e.g., the DB (235) of FIG. 2a) that stores a plurality of vectors and a plurality of image identification information by mapping them.

[0205] According to one embodiment, the electronic device may generate a DB that maps and stores a plurality of vectors and a plurality of image identification information and store it in the memory of the electronic device (e.g., the memory (230) of FIG. 2a).

[0206] According to one embodiment, the electronic device can store in memory a DB in which at least one image identification number is mapped to one vector corresponding to one object.

[0207] According to one embodiment, the electronic device may generate a DB (e.g., DB (235) of FIG. 2a) (e.g., image object table) that maps multiple vectors, for which image embedding has been performed on multiple objects, and multiple identification information (e.g., image id) corresponding to multiple images, and store it in memory (e.g., memory (230) of FIG. 2a). For example, the electronic device may perform image embedding on multiple objects, for example, a couple, a bouquet, a car, and / or a railing, convert them into multiple vectors, and generate a DB (e.g., DB (235) of FIG. 2a) (e.g., image object table) that maps multiple vectors corresponding to the couple, a bouquet, a car, and / or a railing to multiple identification information (e.g., image id) corresponding to multiple images.

[0208] According to one embodiment, when an electronic device identifies a plurality of first objects in which the number of identical objects among a plurality of objects is greater than a specified number, it identifies the plurality of first objects as important objects and can identify one representative object among the plurality of first objects.

[0209] According to one embodiment, when an electronic device identifies a plurality of first objects in which the number of identical objects among a plurality of objects is greater than a specified number, it can generate a cluster for the plurality of first objects, select a central object among the plurality of first objects using a cluster algorithm (e.g., K-means and / or DB-Scan), and identify the central object as a representative object.

[0210] According to one embodiment, the electronic device can use a first AI model (e.g., the first AI model (231a) of FIG. 2b) to embed a representative object and convert it into a vector, identify multiple image identification information corresponding to multiple images including the representative object, and map and store the vector corresponding to the representative object and the multiple image identification information in a DB (e.g., the DB (235) of FIG. 2a).

[0211] According to one embodiment, if the electronic device identifies at least one second object among a plurality of objects in which the number of identical objects is less than or equal to a specified number, it identifies at least one second object as an unimportant object (e.g., an individual object appearing fragmentarily) and may not use at least one second object in a DB (e.g., DB (235) of FIG. 2a) (e.g., an image object table).

[0212] According to one embodiment, the electronic device detects multiple objects from multiple images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a), detects multiple vectors in which the multiple objects are embedded, and then groups the multiple vectors into similar vectors using standardized clustering.

[0213] According to one embodiment, the electronic device may group a plurality of vectors into similar vectors and compare them with a critical cluster number (e.g., the most experimentally meaningful critical cluster number), and based on the comparison result, determine that a second cluster among the groups of similar vectors that is less than or equal to the critical cluster number is invalid and is not used, and determine that a first cluster that is greater than or equal to the critical cluster number is valid.

[0214] According to one embodiment, the electronic device can identify a selected central object as a representative object using a cluster algorithm (e.g., K-means and / or DB-Scan) in a first cluster determined to be valid, and map a plurality of vectors included in the first cluster to a plurality of image identification information to store in a DB (e.g., DB of FIG. 2a (235)).

[0215] FIG. 9 is a flowchart illustrating operations for searching for an image in an electronic device according to one embodiment. Operations for searching for an image may include operations 901 through 923. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0216] In operation 901, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can confirm the input of a first image and a first search term for image search.

[0217] In operation 903, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can identify a first search vector converted into a vector by embedding a first image and a second search vector converted into a vector by embedding a first search term.

[0218] According to one embodiment, the electronic device can verify a first search vector converted into a vector by embedding a first image using a first AI model (e.g., the first AI model (231a) of FIG. 2b).

[0219] According to one embodiment, the first AI model (e.g., the first AI model (231a) of FIG. 2b) may include an image encoder.

[0220] According to one embodiment, the electronic device can use a second AI model (e.g., the second AI model of FIG. 2b (231b)) to identify a second search vector that is converted into a vector by embedding a first search term.

[0221] According to one embodiment, the second AI model (e.g., the second AI model of FIG. 2b (231b)) may include a text encoder.

[0222] In operation 905, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can check a DB for image search.

[0223] According to one embodiment, the electronic device can check a DB (e.g., DB (235) of FIG. 2a) (e.g., image object table) that stores a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images by mapping them in the memory of the electronic device (e.g., memory (230) of FIG. 2a).

[0224] In operation 907, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can compare the similarity between a plurality of vectors stored in a DB and a first search vector.

[0225] In operation 909, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can compare the similarity between a plurality of vectors and a first search vector with a first threshold value.

[0226] In operation 909, if the electronic device fails to identify a vector among a plurality of vectors in which the similarity between the first search vectors is greater than or equal to a first threshold value, in operation 911, the image search operation may be terminated.

[0227] In operation 909, if the electronic device identifies at least one vector among a plurality of vectors in which the similarity between the first search vectors is greater than or equal to a first threshold value, then in operation 913, at least one first candidate vector can be identified.

[0228] According to one embodiment, the electronic device can identify at least one vector among a plurality of vectors stored in the electronic device's DB (e.g., DB (235) of FIG. 2a) as at least one first candidate vector, wherein the similarity between the first search vectors is greater than or equal to a first threshold value.

[0229] In operation 915, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can identify at least one second candidate vector.

[0230] According to one embodiment, the electronic device can identify at least one image identification information mapped to at least one first candidate vector among a plurality of image identification information stored in the electronic device's DB (e.g., DB (235) of FIG. 2a).

[0231] According to one embodiment, the electronic device can identify at least one image corresponding to at least one image identification information (at least one identification information assigned) among a plurality of images stored in the memory of the electronic device (e.g., memory (230) of FIG. 2a).

[0232] According to one embodiment, the electronic device can identify at least one second candidate vector by using a first AI model (e.g., the first AI model (231a) of FIG. 2b) to embed each of at least one image and convert it into a vector.

[0233] In operation 917, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can compare the similarity between at least one second candidate vector and a second search vector.

[0234] In operation 919, if the electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) fails to identify a vector among at least one second candidate vector in which the similarity between the second search vectors is greater than or equal to a second threshold value, the image search operation may be terminated in operation 911.

[0235] In operation 919, if an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) identifies at least one vector among at least one second candidate vector in which the similarity between the second search vectors is greater than or equal to a second threshold value, then in operation 921, at least one final vector can be identified.

[0236] According to one embodiment, the electronic device can identify at least one vector as at least one final vector, wherein the similarity between the second search vectors is greater than or equal to a second threshold value.

[0237] In operation 923, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can display a search result image.

[0238] According to one embodiment, the electronic device can identify at least one image corresponding to at least one final vector among at least one image corresponding to at least one image identification information as a search result image.

[0239] According to one embodiment, the electronic device may display at least one image corresponding to at least one final vector as a search result image through the display of the electronic device (e.g., display (260)).

[0240] According to one embodiment, when the electronic device checks the search result images, it can determine a priority order for displaying the search result images in order of highest similarity based on the similarity with a first search vector and / or a second search vector.

[0241] According to one embodiment, the electronic device may display at least one first search result image in a specified number (e.g., 3) in order of having the highest similarity among the search result images in a first area of ​​the electronic device's display (e.g., display (260) of FIG. 2a).

[0242] According to one embodiment, the electronic device may display at least one second search result image, excluding at least one first search result image among the search result images, as a list in a second area located below the first area of ​​the electronic device's display in order of highest similarity.

[0243] According to one embodiment, when an electronic device checks a search result image, it checks a score using metadata of the search result image and can determine a priority order for displaying the search result images in order of having the highest score based on the score.

[0244] According to one embodiment, the electronic device can determine the number of clicks on an image, the number of shares on an image, and / or the number of searches on an image from the metadata of a search result image, and calculate a score based on <Equation 1>.

[0245] According to one embodiment, the electronic device may display at least one first search result image in a specified number (e.g., 3) in order of having the highest score among the search result images in a first area of ​​the electronic device's display.

[0246] According to one embodiment, the electronic device may display at least one second search result image, excluding at least one first search result image among the search result images, as a list in a second area located below the first area of ​​the electronic device's display in order of having the highest score.

[0247] According to one embodiment, the electronic device may display at least one first search result image of a specified number (e.g., 3) in a first area of ​​the electronic device's display, which has the highest order of similarity to the first search vector and / or the second search vector among the search result images.

[0248] According to one embodiment, the electronic device can check the score using metadata of at least one second search result image excluding at least one first search result image among the search result images, and display at least one second search result image as a list in a second area located below the first area of ​​the electronic device's display in order of highest score based on the score.

[0249] A method for searching for an image in an electronic device (101 in FIG. 1; 201 in FIG. 2a and 2b) according to one embodiment may include, upon confirming the input of a first image and a first search term for image search, verifying a first search vector obtained by converting the first image into a vector using a first AI model, and verifying a second search vector obtained by converting the first search term into a vector using a second AI model. The method according to one embodiment may include an operation of verifying a DB in which a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images are mapped and stored in the memory of the electronic device. The method according to one embodiment may include an operation of verifying at least one first candidate vector among the plurality of vectors stored in the DB of the memory, wherein the similarity to the first search vector is greater than or equal to a first threshold value. The method according to one embodiment may include an operation of verifying at least one image identification information mapped to the at least one first candidate vector among the plurality of image identification information stored in the DB. The method according to one embodiment may include an operation of identifying at least one second candidate vector by converting at least one image corresponding to at least one image identification information into a vector using the first AI model. The method according to one embodiment may include an operation of identifying at least one final vector among the at least one second candidate vector whose similarity to the second search vector is greater than or equal to a second threshold value, and then displaying at least one image corresponding to the at least one final vector as a search result image through the display of the electronic device.

[0250] The method according to one embodiment may include an operation of detecting the plurality of objects in each of the plurality of images stored in the memory. The method according to one embodiment may include an operation of converting the plurality of objects into a plurality of vectors using the first AI. The method according to one embodiment may include an operation of verifying the plurality of identification information corresponding to the plurality of images containing each of the plurality of objects. The method according to one embodiment may include an operation of mapping the plurality of vectors to the plurality of image identification information and storing them in the DB.

[0251] The method according to one embodiment may include an operation of identifying a representative object corresponding to a central object among a plurality of first objects using a cluster algorithm when identifying a plurality of first objects in which the number of identical objects among the plurality of objects is greater than or equal to a specified number. The method according to one embodiment may include an operation of converting the representative object into a vector using the first AI. The method according to one embodiment may include an operation of identifying a plurality of image identification information corresponding to a plurality of images including the representative object. The method according to one embodiment may include an operation of mapping the vector corresponding to the representative object with the plurality of image identification information and storing it in the DB.

[0252] The method according to one embodiment may include an operation of not using the at least one second object in the DB when identifying at least one second object among the plurality of objects in which the number of identical objects is less than or equal to the specified number.

[0253] The method according to one embodiment may include an operation of identifying at least one keyword based on the frequency of search term usage by the user of the electronic device when verifying the input of the first image for the image search. The method according to one embodiment may include an operation of converting the at least one keyword into a vector using the second AI model. The method according to one embodiment may include an operation of displaying at least one image corresponding to the at least one final vector as a search result image through the display when at least one final vector is identified among the at least one second candidate vectors, such that the similarity to the vector corresponding to the at least one keyword is greater than or equal to the second threshold value.

[0254] The method according to one embodiment may include an operation of detecting at least one keyword from a plurality of images corresponding to the plurality of image identification information using a third AI model. The method according to one embodiment may include an operation of mapping and storing the detected at least one keyword to the plurality of vectors and the plurality of image identification information in the DB. The method according to one embodiment may include an operation of verifying a first search vector obtained by converting the first image into a vector using the first AI model when verifying the input of the first image for image search. The method according to one embodiment may include an operation of verifying at least one first candidate vector among the plurality of vectors stored in the DB, wherein the similarity to the first search vector is greater than or equal to a first threshold value. The method according to one embodiment may include an operation of verifying at least one keyword mapped to the at least one first candidate vector in the DB. The method according to one embodiment may include an operation of recommending the at least one keyword as a search term for image search.

[0255] The method according to one embodiment may include an operation of displaying at least one first search result image of a specified number in a first area among the display areas, in the order of highest similarity with the first search vector and / or the second search vector among the search result images. The method according to one embodiment may include an operation of displaying at least one second search result image, excluding the at least one first search result image among the search result images, in a list in the order of highest similarity with the first search vector and / or the second search vector in a second area among the display areas located below the first area.

[0256] The method according to one embodiment may include an operation of checking a score using metadata of the search result image in a first area among the display areas, and displaying at least one first search result image of a specified number having the highest score among the search result images based on the score. The method according to one embodiment may include an operation of checking a score using metadata of at least one second search result image excluding the at least one first search result image among the search result images in a second area located below the first area among the display areas, and displaying at least one second search result image in a list in order of the highest score based on the score.

[0257] The method according to one embodiment may include an operation of displaying at least one first search result image of a specified number having the highest similarity to the first search vector and / or the second search vector among the search result images in a first area among the display areas. The method according to one embodiment may include an operation of checking a score using metadata of at least one second search result image excluding the at least one first search result image among the search result images in a second area located below the first area among the display areas, and displaying the at least one second search result image in order of highest score based on the score as a list containing the at least one second search result image.

[0258] The electronic device according to one embodiment disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment of this document is not limited to the aforementioned devices.

[0259] One embodiment of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0260] The term "module" as used in an embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0261] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101) or electronic device (301)). For example, a processor (e.g., processor (520)) of the machine (e.g., electronic device (301)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0262] According to one embodiment, the method according to one embodiment disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0263] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101 of FIG. 1; 201 of FIG. 2a to 2b), Display (160 in FIG. 1; 260 in FIG. 2a); At least one processor (120 in FIG. 1; 220 in FIG. 2a); and It includes memory for storing instructions (130 in FIG. 1; 230 in FIG. 2a), and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the input of a first image and a first search term for image search is confirmed, a first search vector obtained by converting the first image into a vector is confirmed using a first AI model (231a in FIG. 2b), and a second search vector obtained by converting the first search term into a vector is confirmed using a second AI model (231b in FIG. 2b). Checking a DB that stores a mapping of multiple vectors corresponding to multiple objects and multiple image identification information corresponding to multiple images in the above memory, Identify at least one first candidate vector among the plurality of vectors stored in the DB that has a similarity to the first search vector greater than or equal to a first threshold value, and Identify at least one image identification information mapped to at least one first candidate vector among the plurality of image identification information stored in the above DB, and Using the above first AI model, at least one second candidate vector is identified by converting at least one image corresponding to at least one image identification information into a vector, and An electronic device that, upon identifying at least one final vector among the at least one second candidate vector whose similarity to the second search vector is greater than or equal to a second threshold value, displays at least one image corresponding to the at least one final vector as a search result image through the display.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Detecting the plurality of objects in each of the plurality of images stored in the memory, Using the above first AI, the above plurality of objects are converted into a plurality of vectors, and Identify the plurality of identification information corresponding to the plurality of images containing each of the plurality of objects, and An electronic device that maps the plurality of vectors and the plurality of image identification information and stores them in the DB.

3. In any one of paragraphs 1 to 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When identifying a plurality of first objects among the plurality of objects such that the number of identical objects is greater than or equal to a specified number, a cluster algorithm is used to identify a representative object corresponding to a central object among the plurality of first objects, and Using the above first AI, the above representative object is converted into a vector, and Identifying multiple image identification information corresponding to multiple images including the above representative object, An electronic device that maps a vector corresponding to the above representative object and the above plurality of identification information and stores them in the above DB.

4. In any one of paragraphs 1 to 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that, when identifying at least one second object among the plurality of objects that is identical to the specified number or less, does not use the at least one second object in the DB.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When verifying the input of the first image for the above image search, at least one keyword is verified based on the frequency of search term usage by the user of the electronic device, and Using the above second AI model, the above at least one keyword is converted into a vector, and An electronic device that, upon identifying at least one final vector among the at least one second candidate vectors whose similarity to the vector corresponding to the at least one keyword is greater than or equal to the second threshold value, displays at least one image corresponding to the at least one final vector as a search result image through the display.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Using a third AI model, at least one keyword is detected from a plurality of images corresponding to the plurality of image identification information, and At least one detected keyword is mapped to and stored in the DB as a plurality of image identification information, and When verifying the input of the first image for the above image search, the first search vector obtained by converting the first image into a vector using the first AI model is verified, and Identify at least one first candidate vector among the plurality of vectors stored in the DB that has a similarity to the first search vector greater than or equal to a first threshold value, and Identify at least one keyword mapped to the at least one first candidate vector in the above DB, and An electronic device that recommends at least one keyword as a search term for the image search.

7. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, In a first area among the above display areas, at least one first search result image of a specified number having the highest order of similarity with the first search vector and / or the second search vector among the search result images is displayed, and An electronic device that displays, in a second area located below the first area among the areas of the above display, at least one second search result image among the search result images, excluding at least one first search result image, in a list in order of highest similarity with the first search vector and / or the second search vector.

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, In a first area among the above display areas, a score is checked using metadata of the search result image, and based on the score, at least one first search result image of a specified number having the highest score among the search result images is displayed. An electronic device that checks a score using metadata of at least one second search result image, excluding at least one first search result image among the search result images, in a second area located lower than the first area among the areas of the display, and displays the at least one second search result image in a list in order of highest score based on the score.

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, In a first area among the above display areas, at least one first search result image of a specified number having the highest order of similarity with the first search vector and / or the second search vector among the search result images is displayed, and An electronic device that checks a score using metadata of at least one second search result image, excluding at least one first search result image among the search result images, in a second area located lower than the first area among the areas of the display, and displays the at least one second search result image in order of highest score based on the score.

10. A method for retrieving an image in an electronic device (101 of FIG. 1; 201 of FIG. 2a to 2b), When the input of a first image and a first search term for image search is confirmed, the operation of confirming a first search vector obtained by converting the first image into a vector using a first AI model, and confirming a second search vector obtained by converting the first search term into a vector using a second AI model; An operation to check a DB storing a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images in the memory of the electronic device; An operation to identify at least one first candidate vector among the plurality of vectors stored in the DB of the memory, wherein the similarity to the first search vector is greater than or equal to a first threshold value; An operation to identify at least one image identification information mapped to at least one first candidate vector among the plurality of image identification information stored in the DB; The operation of identifying at least one second candidate vector, which is obtained by converting at least one image corresponding to at least one image identification information into a vector using the above first AI model; and A method comprising, when identifying at least one final vector among the at least one second candidate vector that has a similarity to the second search vector greater than or equal to a second threshold value, displaying at least one image corresponding to the at least one final vector as a search result image through the display of the electronic device.

11. In Paragraph 10, An operation of detecting the plurality of objects in each of the plurality of images stored in the memory; An operation of converting the plurality of objects into a plurality of vectors using the first AI above; An operation to verify the plurality of identification information corresponding to the plurality of images, each containing the plurality of objects; and A method further comprising the operation of mapping the plurality of vectors and the plurality of image identification information and storing them in the DB.

12. In any one of paragraphs 10 to 11, When identifying a plurality of first objects in which the number of identical objects among the plurality of objects is greater than a specified number, an operation of identifying a representative object corresponding to a central object among the plurality of first objects using a cluster algorithm; An operation to convert the representative object into a vector using the first AI above; An operation to identify a plurality of image identification information corresponding to a plurality of images including the above representative object; and A method further comprising the operation of mapping a vector corresponding to the above representative object and the above plurality of identification information and storing them in the above DB.

13. In any one of paragraphs 10 to 12, A method further comprising an operation of not using the at least one second object in the DB when identifying at least one second object among the plurality of objects that is identical to the specified number or less.

14. In any one of paragraphs 10 through 13, When verifying the input of the first image for the above image search, the operation of verifying at least one keyword based on the frequency of search term usage by the user of the electronic device; Using the above second AI model, the operation of converting the above at least one keyword into a vector; and A method further comprising, when identifying at least one final vector among the at least one second candidate vectors such that the similarity with the vector corresponding to the at least one keyword is greater than or equal to the second threshold value, displaying at least one image corresponding to the at least one final vector as a search result image through the display.

15. In a non-volatile storage medium storing instructions, said instructions are configured to cause said electronic device to perform at least one operation when executed by said electronic device, said at least one operation being, When the input of a first image and a first search term for image search is confirmed, the operation of confirming a first search vector obtained by converting the first image into a vector using a first AI model, and confirming a second search vector obtained by converting the first search term into a vector using a second AI model; An operation to check a DB storing a plurality of vectors corresponding to a plurality of objects and a plurality of image identification information corresponding to a plurality of images in the memory of the electronic device; An operation to identify at least one first candidate vector among the plurality of vectors stored in the DB of the memory, wherein the similarity to the first search vector is greater than or equal to a first threshold value; An operation to identify at least one image identification information mapped to at least one first candidate vector among the plurality of image identification information stored in the DB; The operation of identifying at least one second candidate vector, which is obtained by converting at least one image corresponding to at least one image identification information into a vector using the above first AI model; and A storage medium comprising the operation of, when identifying at least one final vector among the at least one second candidate vector whose similarity to the second search vector is greater than or equal to a second threshold value, displaying at least one image corresponding to the at least one final vector as a search result image through the display of the electronic device.

Images