Method for generating response and electronic device for performing same

Abstract

A method performed by an electronic device may comprise the operations of: generating first data of a short-term memory level by performing first processing on original data acquired by the electronic device; generating second data of a long-term memory level by performing second processing on at least a portion of the first data; receiving a query of a user; determining a target session for generating a response to the query on the basis of context information of the query; determining session memory data corresponding to the target session including at least one of a portion of the first data or a portion of the second data on the basis of the context information; generating a response to the query on the basis of at least the session memory data; and outputting the response.

Description

Method for generating a response and electronic device for performing the same

[0001] One embodiment disclosed in this document relates to a method for generating a response to a user's query, an electronic device for performing the method, and a storage medium. Below, a technique for generating a response regarding an environment image acquired by an electronic device using a trained model is disclosed.

[0002] Extended reality technologies, such as virtual reality, augmented reality, and mixed reality, which utilize computer graphics technology, are being developed. Virtual reality technology allows users to perceive a virtual space constructed by a computer that does not exist in the real world as reality. Augmented reality or mixed reality technology enables the integration of the real world and the virtual world by overlaying computer-generated information onto the real world, and allows for real-time interaction with the user.

[0003] According to the VST (video see-through) method, after an image of the physical environment is captured using a camera, the captured image can be displayed on a display screen. At this time, digital information can be provided superimposed on the captured image. Meanwhile, according to the OST (optical see-through) method, while the user directly views the physical environment using a transparent display or lens, digital information can be provided superimposed on the physical environment.

[0004] Meanwhile, retrieval-augmented generation (RAG) is a process that allows a large language model (LLM) to reference a trusted knowledge base outside of the LLM's training data sources before generating a response. Based on a user's query, relevant documents can be retrieved from external data sources existing in various forms, such as application program interfaces (APIs), databases, or document repositories. The LLM can then generate a final response using the documents retrieved along with the user's query as input.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] According to one embodiment, the electronic device may include at least one processor comprising processing circuitry. The electronic device may include a memory comprising one or more storage media for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of generating first data at a short-term memory level by performing a first processing on original data acquired by the electronic device. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of generating second data at a long-term memory level by performing a second processing on at least a portion of the first data. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of receiving a user query. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of determining a target session for generating a response to the query based on context information of the query. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of determining session memory data corresponding to the target session, which includes at least one of the part of the first data or the part of the second data, based on the context information.When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be made to perform an operation of generating a response to the query based on at least the session memory data. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be made to perform an operation of outputting the response.

[0007] According to one embodiment, an electronic device may include at least one processor comprising processing circuitry. The electronic device may include a memory comprising one or more storage media for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of acquiring original data. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of generating first data at a short-term memory level by performing a first processing on the original data. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of generating second data at a long-term memory level by performing a second processing on at least a portion of the first data. The operation of generating the second data at the long-term memory level by performing the second processing on at least a portion of the first data may include an operation of determining the importance of each portion of the first data based on a timestamp regarding each portion of the first data and the frequency with which each portion of the first data is utilized for search augmentation to generate a response for a user of the electronic device. The operation of generating the second data at the long-term memory level by performing the second processing on at least a portion of the first data may include an operation of generating the second data by performing the second processing on at least a portion of the first data where the corresponding importance satisfies a predetermined condition regarding the second processing.

[0008] A method performed by an electronic device according to one embodiment may include an operation of generating first data at a short-term memory level by performing a first processing on original data acquired by the electronic device. The method may include an operation of generating second data at a long-term memory level by performing a second processing on at least a portion of the first data. The method may include an operation of receiving a user query. The method may include an operation of determining a target session for generating a response to the query based on context information of the query. The method may include an operation of determining session memory data corresponding to the target session containing at least one of a portion of the first data or a portion of the second data based on the context information. The method may include an operation of generating a response to the query based on at least the session memory data. The method may include an operation of outputting the response.

[0009] A method performed by an electronic device according to one embodiment may include an operation of acquiring original data. The method may include an operation of generating first data at a short-term memory level by performing a first processing on the original data. The method may include an operation of generating second data at a long-term memory level by performing a second processing on at least a portion of the first data. The operation of generating the second data at a long-term memory level by performing the second processing on at least a portion of the first data may include an operation of determining the importance of each portion of the first data based on a timestamp regarding each portion of the first data and the frequency with which each portion of the first data is utilized for search augmentation to generate a response for a user of the electronic device. The operation of generating the second data at a long-term memory level by performing the second processing on at least a portion of the first data may include an operation of generating the second data by performing the second processing on at least a portion of the first data where the corresponding importance satisfies a predetermined condition regarding the second processing.

[0010] According to one embodiment, a non-transient computer-readable recording medium may store one or more programs including instructions. When the instructions are executed individually or collectively by at least one processor of an electronic device, the electronic device may be enabled to perform an operation of generating first data at a short-term memory level by performing a first processing on original data acquired by the electronic device. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of generating second data at a long-term memory level by performing a second processing on at least a portion of the first data. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of receiving a user query. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of determining a target session for generating a response to the query based on context information of the query. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to perform an operation of determining session memory data corresponding to the target session, which includes at least one of the first data portion or the second data portion, based on the context information. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to perform an operation of generating a response to the query based on at least the session memory data.When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be made to perform the operation of outputting the response.

[0011] According to one embodiment, a non-transient computer-readable recording medium may store one or more programs including instructions. When the instructions are executed individually or collectively by at least one processor of an electronic device, the electronic device may be enabled to perform an operation of acquiring original data. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of generating first data at a short-term memory level by performing a first processing on the original data. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to perform an operation of generating second data at a long-term memory level by performing a second processing on at least a portion of the first data. The operation of generating the second data at the long-term memory level by performing the second processing on at least a portion of the first data may include an operation of determining the importance of each portion of the first data based on a timestamp regarding each portion of the first data and the frequency with which each portion of the first data is utilized for search augmentation to generate a response for a user of the electronic device. The operation of generating the second data at the long-term memory level by performing the second processing on at least a portion of the first data may include an operation of generating the second data by performing the second processing on at least a portion of the first data where the corresponding importance satisfies a predetermined condition regarding the second processing.

[0012] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0013] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0014] FIG. 2 illustrates examples of optical see-through devices according to various embodiments.

[0015] FIGS. 3a and 3b are drawings showing examples of the front and rear of an electronic device according to various embodiments.

[0016] FIG. 4a is a drawing illustrating an artificial intelligence system according to one embodiment.

[0017] FIG. 4b is a block diagram of an artificial intelligence (AI) framework according to one embodiment.

[0018] FIG. 5 schematically illustrates the operation method of a response system according to one embodiment.

[0019] FIG. 6 is a flowchart of a method for generating a response according to one embodiment.

[0020] FIG. 7 is a flowchart of a method for determining a target session according to one embodiment.

[0021] FIG. 8 is a flowchart of a method for determining session memory data according to one embodiment.

[0022] FIG. 9 is a flowchart of a method for determining session memory data according to one embodiment.

[0023] FIGS. 10a and FIGS. 10b are drawings illustrating a method of providing a response according to one example, respectively.

[0024] FIG. 11 is a flowchart of a method for generating and managing data at the short-term and long-term memory level according to one embodiment.

[0025] FIG. 12 is a flowchart of a method for converting data at a short-term memory level to data at a long-term memory level according to one embodiment.

[0026] FIGS. 13a, FIGS. 13b, and FIGS. 13c are drawings illustrating a method for refining data at the short-term and long-term memory level according to one example, respectively.

[0027] FIG. 14 is a flowchart of a method for refining data at the long-term memory level according to one embodiment.

[0028] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0029] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0030] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0031] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0032] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0033] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0034] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0035] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0039] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0040] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0041] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0045] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0047] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0050] An antenna module (197) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0051] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0052] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] FIG. 2 illustrates examples of optical see-through devices according to various embodiments.

[0055] An electronic device (201) (e.g., the electronic device (101) of FIG. 1) may include at least one of a display (e.g., the display module (160) of FIG. 1), a vision sensor, a light source (230a, 230b), an optical element, or a substrate. According to one embodiment, the display of the electronic device (201) is transparent and may provide an image through the transparent display. According to one embodiment, the electronic device (201) may include a transparent member and a display connected to the transparent member. A user may look at an object placed in physical space (e.g., an object in the real world) through the transparent member. An electronic device (201) that allows light reflected from an object placed in physical space to pass through a transparent configuration (e.g., a transparent display, a transparent member separate from the display) and provides an image through the display may be referred to as an optical see-through device (OST device).

[0056] The display may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0057] In one embodiment, where the display is composed of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the electronic device (201) may include a light source (230a, 230b) that irradiates light onto a screen output area of ​​the display (e.g., a screen display portion (215a, 215b)). In another embodiment, where the display can generate light on its own, for example, where it is composed of an organic light-emitting diode or a micro LED, the electronic device (201) may provide a virtual image of good quality to the user without including a separate light source (230a, 230b). In one embodiment, if the display is implemented as an organic light-emitting diode or a micro LED, the light source (230a, 230b) is unnecessary, so the electronic device (201) can be made lighter.

[0058] Referring to FIG. 2, the electronic device (201) may include a display, a first transparent member (225a) and / or a second transparent member (225b), and the user may use the electronic device (201) while wearing it on their face. The first transparent member (225a) and / or the second transparent member (225b) may be formed from a glass plate, a plastic plate, or a polymer, and may be made transparent or translucent. According to one embodiment, the first transparent member (225a) may be positioned facing the user's right eye, and the second transparent member (225b) may be positioned facing the user's left eye. The display may include a first display (205) that outputs a first image (e.g., right image) corresponding to the first transparent member (225a) and a second display (210) that outputs a second image (e.g., left image) corresponding to the second transparent member (225b). According to one embodiment, when each display is transparent, each display and the transparent member may be positioned to face the user's eyes to form a screen display unit (215a, 215b).

[0059] In one embodiment, light emitted from a display (205, 210) may be guided along a light path to a waveguide through an input optical member (220a, 220b). Light traveling within the waveguide may be guided toward the user's eye through an output optical member (e.g., an output grating region). Screen display units (215a, 215b) may be determined based on the light emitted toward the user's eye.

[0060] For example, light emitted from the display (205, 210) can be reflected by the grating region of the waveguide formed in the input optical member (220a, 220b) and the screen display portions (215a, 215b) and transmitted to the user's eye.

[0061] The optical element may include at least one of a lens or an optical waveguide.

[0062] The lens can adjust the focus so that the screen output to the display can be seen by the user's eyes. The lens may include, for example, at least one of a Fresnel lens, a pancake lens, or a multichannel lens.

[0063] An optical waveguide can transmit an image ray generated from a display to the user's eye. For example, the image ray may represent a ray of light emitted by a light source (230a, 230b) that has passed through the screen output area of ​​the display. The optical waveguide may be made of glass, plastic, or polymer. The optical waveguide may include a nano-pattern formed on some internal or external surface, for example, a polygonal or curved grating structure.

[0064] The vision sensor may include at least one of a camera sensor or a depth sensor.

[0065] The first camera (265a, 265b) is a recognition camera and may be used for 3DoF and 6DoF head tracking, hand detection, hand tracking, and spatial recognition. The first camera (265a, 265b) may primarily include a GS (global shutter) camera. Since stereo cameras are required for head tracking and spatial recognition, the first camera (265a, 265b) may include two or more GS cameras. A GS camera may have superior performance compared to a RS (rolling shutter) camera in terms of detecting fast hand movements and fine movements such as fingers, and tracking movements. For example, a GS camera may have low image blur. The first camera (265a, 265b) can capture image data used for 6DoF spatial recognition and SLAM functions through depth capture. In addition, a user gesture recognition function can be performed based on image data captured by the first camera (265a, 265b).

[0066] The second camera (270a, 270b) is an ET (eye tracking) camera and can be used to capture image data for detecting and tracking the user's pupils. The second camera (270a, 270b) can track the user's eyes, that is, the user's gaze, using light (e.g., infrared light) output from a display. The second camera (270a, 270b) may be an eye-tracking camera that collects information to position the center of a virtual image projected onto the electronic device (201) according to the direction in which the wearer's pupils of the electronic device (201) gaze. The second camera (270a, 270b) may also include a GS camera to detect the pupils and track rapid pupil movements. The ET camera may also be installed for the left eye and the right eye, respectively, and the same camera performance and specifications may be used for each. The second camera (270a, 270b) may include a gaze tracking sensor. The eye tracking sensor may be included inside the second camera (270a, 270b). Infrared light output from the display (205, 210) may be transmitted to the user's eye as infrared reflected light by a half mirror. The eye tracking sensor may detect infrared transmitted light reflected from the user's eye. The second camera (270a, 270b) may track the user's eye, that is, the user's gaze, based on the detection result of the eye tracking sensor.

[0067] The third camera (245) may be a camera for shooting. The third camera (245) may include a high-resolution camera for capturing images of HR (high resolution) or PV (photo video). The third camera (245) may include a color camera equipped with functions for obtaining high-quality images, such as AF function and optical image stabilization (OIS). The third camera (245) may be a GS camera or an RS camera.

[0068] The fourth camera (e.g., the face recognition camera (325, 326) of FIGS. 3a and 3b below) is a face recognition camera, and the FT (face tracking) camera can be used to detect and track the user's facial expressions.

[0069] A depth sensor (not shown) may represent a sensor that senses information for determining the distance to an object, such as Time of Flight (TOF). TOF is a technology that measures the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser). A depth sensor based on TOF technology emits a signal from a transmitter and measures the signal at a receiver, and can measure the flight time of the signal.

[0070] A light source (230a, 230b) (e.g., an illumination module) may include a device (e.g., a light emitting diode) that emits light of various wavelengths. The illumination module may be attached in various locations depending on the application. In one use case, a first illumination module (e.g., an LED device) attached around the frame of an augmented reality glasses device may emit light to assist in gaze detection when tracking eye movements with an ET camera. The first illumination module may, for example, include an IR LED of infrared wavelength. In another use case, a second illumination module (e.g., an LED device) may be attached adjacent to a camera mounted around a hinge (240a, 240b) connecting the frame and the temple, or around a bridge connecting the frame. The second illumination module may emit light to supplement ambient brightness when the camera is taking pictures. If subject detection is not easy in a dark environment, the second illumination module may emit light.

[0071] A substrate (235a, 235b) (e.g., a printed circuit board (PCB)) can support the aforementioned components.

[0072] A printed circuit board (PCB) may be placed on the temple of the glasses. The FPCB may transmit electrical signals to each module (e.g., camera, display, audio module, sensor) and other printed circuit boards. According to one embodiment, at least one printed circuit board may be in the form of a first board, a second board, and an interposer disposed between the first board and the second board. Electrical signals may be transmitted to each module and other printed circuit boards.

[0073] Other components may include at least one of a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), a third microphone (250c)), a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)), a battery (260), an antenna, or a sensor (e.g., an accelerometer, a gyroscope, or a touch sensor).

[0074] FIGS. 3a and 3b are drawings showing examples of the front and rear of an electronic device according to various embodiments.

[0075] FIG. 3a is an external view of the electronic device (301) viewed from a first direction (①), and FIG. 3b is an external view of the electronic device (301) viewed from a second direction (②). When a user wears the electronic device (301), the external view seen by the user's eyes may be FIG. 3b.

[0076] Referring to FIG. 3a, according to various embodiments, an electronic device (301) (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) may provide a service that provides an extended reality (XR) experience to a user. For example, XR or XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), and / or mixed reality (MR).

[0077] According to one embodiment, the electronic device (301) may refer to a head-mounted device or a head-mounted display worn on the head of a user, but may be configured in the form of at least one of glasses, goggles, a helmet, or a hat. The electronic device (301) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through the glass when worn, or a VST (video see-through) type configured to block external light so that light emitted from the display reaches the user's eyes when worn, but external light does not reach the user's eyes.

[0078] According to one embodiment, an electronic device (301) may be worn on the head of a user to provide the user with images related to an extended reality (XR) service. For example, the electronic device (301) may provide XR content (hereinafter referred to as XR content images) that outputs at least one virtual object superimposed on an area determined to be a display area or the user's field of view (FoV). According to one embodiment, XR content may refer to images related to real space acquired through a camera (e.g., a camera for shooting) or images or videos that appear to have at least one virtual object superimposed on a virtual space. According to one embodiment, the electronic device (301) may provide XR content based on a function being performed on the electronic device (301) and / or a function being performed on one or more external electronic devices (e.g., the electronic devices (102, 104) of FIG. 1, the server (108) of FIG. 1).

[0079] According to one embodiment, the electronic device (301) is at least partially controlled by an external electronic device (e.g., the electronic device (102, 104) of FIG. 1), and at least one function may be performed under the control of the external electronic device, but at least one function may also be performed independently.

[0080] Referring to FIG. 3a, a vision sensor may be disposed on a first surface of the housing of the main body (310) of the electronic device (301). The vision sensor may include cameras (e.g., cameras for a second function (311, 312), cameras for a first function (315)) and / or a depth sensor (317) for acquiring information related to the surrounding environment of the electronic device (301).

[0081] In one embodiment, the second function cameras (311, 312) can acquire images related to the surrounding environment of the electronic device (301). The first function cameras (315) can acquire images while the wearable electronic device is worn by a user. The first function cameras (315) can be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The first function cameras (315) can be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. In one embodiment, the second function cameras (311, 312) may be used for hand detection and tracking, and user gestures.

[0082] In one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from the subject, and may be used for determining the distance to the object, such as time of flight (TOF). Instead of or in addition to the depth sensor (317), cameras (311, 312, 315, 316) may determine the distance to the object.

[0083] Referring to FIG. 3b, a face recognition camera (325, 326) and / or a display (321) (and / or a lens) may be disposed on the second surface (320) of the main body (310) housing.

[0084] In one embodiment, a face recognition camera (325, 326) adjacent to the display may be used to recognize the user's face or to recognize and / or track both of the user's eyes.

[0085] In one embodiment, the display (321) (and / or lens) may be disposed on a second surface (320) of the electronic device (301). In one embodiment, the electronic device (301) may not include some of the plurality of cameras (315). Although not illustrated in FIGS. 3a and 3b, the electronic device (301) may further include at least one of the configurations illustrated in FIG. 2.

[0086] According to one embodiment, the electronic device (301) may include a main body (310) that implements at least some of the components of FIG. 1, a display (321) (e.g., the display module (160) of FIG. 1) disposed in a first direction (①) of the main body (310), a first function camera (e.g., a recognition camera) (315) disposed in a second direction (②) of the main body (310), a second function camera (e.g., a shooting camera) (311, 312) disposed in a second direction (②), a third function camera (e.g., a gaze tracking camera) (328) disposed in a first direction (①), a fourth function camera (e.g., a face recognition camera) (325, 326) disposed in a first direction (①), a depth sensor (317) disposed in a second direction (②), and a touch sensor (313) disposed in a second direction (②). Although not shown in the drawing, the main body (310) may include a memory (e.g., memory (130) of FIG. 1) and a processor (e.g., processor (120) of FIG. 1) inside, and may further include other components shown in FIG. 1.

[0087] According to one embodiment, the display (321) may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0088] In one embodiment, if the display (321) is one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the electronic device (301) may include a light source that irradiates light onto the screen output area of ​​the display (321). In another embodiment, if the display (321) can generate light itself, for example, if the electronic device (301) is one of an organic light-emitting diode or a micro LED, the electronic device (301) can provide a user with high-quality XR content images without including a separate light source. In one embodiment, if the display (321) is implemented as an organic light-emitting diode or a micro LED, a light source is unnecessary, so the electronic device (301) can be made lighter.

[0089] According to one embodiment, the electronic device (301) may include a plurality of cameras. For example, the cameras may include a first functional camera (e.g., a recognition camera) (315) positioned in the second direction (②) of the main body (310), a second functional camera (e.g., a shooting camera) (311, 312) positioned in the second direction (②), a third functional camera (e.g., a gaze tracking camera) (328) positioned in the first direction (①) and / or a fourth functional camera (e.g., a face recognition camera) (325, 326) positioned in the first direction (①), but may further include cameras of other functions not illustrated.

[0090] The first functional camera (e.g., recognition camera) (315) may be used for detecting user movement or user gesture recognition functions. The first functional camera (315) may support at least one of head tracking, hand detection and hand tracking, and spatial recognition. For example, the first functional camera (315) may primarily use a GS (global shutter) camera, which has superior performance compared to an RS (rolling shutter) camera, to detect hand movements and fine finger movements and to track movements, and may be composed of a stereo camera including two or more GS cameras for head tracking and spatial recognition. The first functional camera (315) may perform SLAM (simultaneous localization and mapping) functions to recognize information related to the surrounding space (e.g., location and / or orientation) through spatial recognition for 6DoF and depth capture.

[0091] A second-function camera (e.g., a camera for shooting) (311, 312) can be used to capture the outside and generate an image or video corresponding to the outside and transmit it to a processor (e.g., the processor (120) of FIG. 1). The processor can display the image received from the second-function camera (311, 312) on a display (321). The second-function camera (311, 312) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. For example, the second-function camera (311, 312) may include a color camera equipped with functions for obtaining high-quality images, such as AF (auto focus) and shake correction (OIS (optical image stabilizer)), but is not limited thereto, and the second-function camera (311, 312) may also include a GS camera or an RS camera.

[0092] A third-function camera (e.g., eye-tracking camera) (328) may be placed in the display (321) (or inside the main body) such that the camera lens faces the user's eyes when the user is equipped with the electronic device (301). The third-function camera (328) may be used for detecting and tracking the pupils (ET: eye tracking). A processor may determine the direction of gaze by tracking the movement of the user's left and right eyes in the image received from the third-function camera (328). By tracking the position of the pupils in the image, the processor may position the center of the XR content image displayed in the display area according to the direction the pupils are gazing. As an example, a GS camera may be used for the third-function camera (328) to detect the pupils and track pupil movements. The third-function camera (328) may be installed for the left and right eyes respectively, and the same camera performance and specifications may be used for each.

[0093] The fourth functional camera (e.g., face recognition camera) (325, 326) can be used to detect and track (FT: face tracking) the user's facial expression when the user is wearing the electronic device (301).

[0094] According to one embodiment, the electronic device (301) may include a lighting unit (e.g., LED) (not shown) as an auxiliary means for the cameras. For example, the third function camera (325) may use lighting included in the display so that emitted light (e.g., IR LED of infrared wavelength) is directed toward both eyes of the user as an auxiliary means to facilitate gaze detection when tracking eye movements. As another example, the second function cameras (311, 312) may further include a lighting unit (e.g., flash) as an auxiliary means to supplement ambient brightness when shooting outdoors.

[0095] According to one embodiment, a depth sensor (or depth camera) (317) may be used for determining the distance to an object (e.g., object) such as time of flight (TOF). Time of flight (TOF) is a technique for measuring the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser), in which a transmitter transmits a signal, a receiver measures the signal, and the distance to the object can be measured based on the flight time of the signal.

[0096] According to one embodiment, the touch sensor (313) may be positioned in the second direction (②) of the main body (310). For example, when a user wears the electronic device (301), the user's eyes may look toward the first direction (①) of the main body. The touch sensor (313) may be implemented as a single type or a type separated into left and right sides depending on the shape of the main body (310), but is not limited thereto. For example, if the touch sensor (313) is implemented as a type separated into left and right sides as shown in FIG. 3a, when a user wears the electronic device (301), the first touch sensor (313a) may be positioned at the user's left eye position as in the fourth direction (④), and the second touch sensor (313b) may be positioned at the user's right eye position as in the third direction (③).

[0097] The touch sensor (313) can recognize touch input in at least one of, for example, capacitive, pressure-sensitive, infrared, or ultrasonic methods. For example, the capacitive touch sensor (313) may be capable of recognizing physical touch (or contact) input or hovering input (or proximity) of an external object. According to some embodiments, the electronic device (301) may use a proximity sensor (not shown) to enable proximity recognition of an external object.

[0098] According to one embodiment, the touch sensor (313) has a two-dimensional surface and can transmit touch data (e.g., touch coordinates) of an external object (e.g., user finger) that contacts the touch sensor (313) to a processor (e.g., processor (120) of FIG. 1). The touch sensor (313) can detect a hovering input for an external object (e.g., user finger) that approaches within a first distance from the touch sensor (313), or detect a touch input that touches the touch sensor (313).

[0099] According to one embodiment, when an external object touches the touch sensor (313), the touch sensor (313) may provide two-dimensional information about the contact point to the processor (120) as "touch data." The touch data may be described as "touch mode." When an external object is located within a first distance from the touch sensor (313) (or is in close proximity, hovering above the touch sensor), the touch sensor (313) may provide hovering data to the processor (120) regarding the time or location of hovering around the touch sensor (313). The hovering data may be described as "hovering mode / closeness mode."

[0100] According to one embodiment, the electronic device (301) can acquire hovering data using at least one of the touch sensor (313), a proximity sensor (not shown) and / or a depth sensor (317) to generate information regarding the distance, location, or time between the touch sensor (313) and an external object.

[0101] According to one embodiment, the interior of the main body (310) may include a processor (e.g., the processor (120) of FIG. 1) and a memory (e.g., the memory (130) of FIG. 1).

[0102] Memory can store various instructions that can be executed by the processor. Instructions may include arithmetic and logical operations, data movement, or control instructions such as input / output that can be recognized by the processor. Memory may include volatile memory (e.g., volatile memory (132) of FIG. 1) and non-volatile memory (e.g., non-volatile memory (134) of FIG. 1) and may store various data temporarily or permanently.

[0103] The processor may be configured to be operatively, functionally, and / or electrically connected to each component of the electronic device (301) and capable of performing operations or data processing regarding the control and / or communication of each component. The operations performed by the processor may be executed by instructions that are stored in memory and, at execution, cause the processor to operate.

[0104] Hereinafter, although there are no limitations on the computation and data processing functions that the processor can implement on the electronic device (301), a series of operations related to XR content service functions will be described. The operations of the processor described below can be performed by executing instructions stored in memory.

[0105] According to one embodiment, the processor can create a virtual object based on virtual information based on image information. The processor can output a virtual object related to an XR service along with background space information through a display (321). For example, the processor can acquire image information by capturing an image related to a real space corresponding to the field of view of a user wearing an electronic device (301) through a second function camera (311, 312), or can create a virtual space for a virtual environment. For example, the processor can control the display (321) to display XR content (hereinafter referred to as the XR content screen) such that at least one virtual object is superimposed on an area determined to be a field of view or a user's field of view (FoV).

[0106] According to one embodiment, the electronic device (301) may have a form factor for being worn on a user's head. The electronic device (301) may further include a strap and / or a wearing member for being secured on a part of the user's body. The electronic device (301) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0107] FIG. 4a is a drawing illustrating an artificial intelligence system according to one embodiment.

[0108] Referring to FIG. 4a, the generative AI system (400) may include a user interface (410), an AI framework (420), a generative AI model (430), a knowledge repository (440), and an application / service module (450). These components may be operated on one or more of an electronic device (101), an external electronic device (102 or 104), or a server (108). For example, the user interface (410) and the AI ​​framework (420) may be operated on the electronic device (101), and the knowledge repository (440) and the generative AI model (430) may be operated on the server (108).

[0109] According to one embodiment, the user interface (410) may receive user input (e.g., user query). User input may be received in the form of text, images, voice (e.g., natural language), video, menu selection, or a combination thereof. The user interface (410) may include various context information (e.g., running application or user location) related to the generative artificial intelligence system (400) at the time the user input is received, in addition to or instead of the user input. The user interface (410) may provide the user input or the context information to the AI ​​framework (420) and provide the result of processing therefrom to the user, for example, through the AI ​​framework (420). According to one embodiment, in addition to user input, the electronic device may provide context information obtained using information included on the screen to the AI ​​framework (420). The result may be provided in the form of text, images, voice, video, actions requested by the user (e.g., execution of a specified function or app), or a combination thereof.

[0110] According to one embodiment, the AI ​​framework (420) can identify (e.g., estimate) a user intent based on at least some user input or context information received from a user interface (410), control each of the relevant modules (e.g., 421, 423, or 425) to perform a function or action corresponding to the identified user intent, and coordinate cooperation between two or more modules. The AI ​​framework (420) may include a prompt design module (421), an API / Plug-in management module (423), and an output modification module (425), as illustrated in FIG. 2.

[0111] According to one embodiment, the prompt design module (421) can generate a prompt to be input to a generative AI model (430) based at least partially on user input or context information received from a user interface (410). For example, the prompt design module (421) can generate a prompt using user preferences, a prompt library, or prompt examples stored in a knowledge repository (440) based at least partially on user input or context information.

[0112] According to one embodiment, the API-Plug-in management module (423) may communicate, for example, via an API, with various resources (e.g., a knowledge repository (440)) that provide said additional information when there is a request for said additional information in relation to user input. Additionally or generally, when a specified action (e.g., a function, app, or service) is performed in response to said user input, the API-Plug-in management module (423) may request the application / service module (450) to perform said specified action via a corresponding API. The API-Plug-in management module (423) may provide information obtained from the knowledge repository (440), the application / service module 450, or another external resource to the prompt design module (421). That obtained information may be used by the prompt design module (421) to generate a prompt together with the user input, or provided to a generative AI model (430).

[0113] According to one embodiment, the output modification module (425) can fine-tune the results obtained through the generative AI model (430) as at least part of the response to user input (e.g., user query). For example, the output modification module (425) can determine whether the content of the response obtained through the generative AI model (430) is appropriate as a response to a request made by the user input. For example, the output modification module (425) can determine the degree of relevance, degree of bias (e.g., political or social bias), or degree of harmfulness (e.g., sexual or profanity) of the difference between the response obtained through the generative AI model (430) and the user input. Additionally or generally, the output modification module (425) can request that additional AI processing be performed on the obtained response, or provide the user with a hint to avoid unwanted output. For example, the response can be obtained again through the generative AI model (430) by generating additional prompts through the prompt design module.

[0114] According to one embodiment, the generative AI model (430) may form at least part of an artificial intelligence neural network and may include a model that generates images or a model that generates language. The image generation model may include, for example, a generative adversarial network (GAN), a variational auto encoder (VAE), or a diffusion-based model using a VAE and a transformer. The language generation model may include, for example, a large language model (LLM), a large multimodal model (LMM), a large vision model (LVM), or a large action model (LAM). The LAM may automatically generate actions for an environment (e.g., a robot, a car, an electronic device (101), or a program (140)). Additionally, for at least some AI models (e.g., LLM), there may be a low-rank adaptation (LoRA) adapter fine-tuned for, for example, a specific task or a specific situation.

[0115] FIG. 4b is a block diagram of an AI framework according to one embodiment.

[0116] FIG. 4b illustrates an AI framework (420) having on-device AI processing capabilities according to one embodiment. In this case, the AI ​​framework (420) may generate and learn a response to the user input using resources within the device, instead of sending the user input received through a user interface (410) operating on the same device (e.g., electronic device (101)) to a generative AI model (430) operating on an external device (e.g., server (108)), or additionally. Referring to FIG. 4b, the AI ​​framework (420) may include a cross-application action module (461), a personal data managing module (463), an on-device AI model (465), and an orchestration module (467).

[0117] According to one embodiment, the cross-application action module (461) determines one or more additional applications required for the operation of an executed application (e.g., an assistant app) and may connect or suggest operations between the app and at least one additional application, or between a plurality of additional applications. For example, the cross-application action module (461) may execute one or more additional applications to be used to respond to a user request through the assistant app sequentially or at least partially and simultaneously. Additionally, the cross-application action module (461) may communicate with the additional applications so that the result of the execution of one additional application (e.g., content) can be shared with other additional applications.

[0118] According to one embodiment, the personal data management module (463) may provide personal information (e.g., schedule, contact, or message information) about the user of the application (e.g., assistant app) or the additional application running on the device (e.g., electronic device 101) or other related individuals (e.g., family or friends) to another module of the AI ​​framework (420) or a related module (e.g., generative AI model 430) operating on another device.

[0119] According to one embodiment, the on-device AI model (465) may include at least one model among one or more AI models (e.g., GAN, VAE, LLM, LMM, LVM, or LAM) operated on an external device (e.g., server (108)) or a corresponding lightweight AI model. Additionally, for said model or said lightweight model, there may be, for example, a LoRA adapter.

[0120] According to one embodiment, the orchestration module (467) may select one or more AI models to be used to obtain a response to user input (e.g., user query). For example, the orchestration module (467) may select one or more AI models among an on-device AI model (465), an AI model operating on an external device (e.g., server (108)) (e.g., generative AI model (430)), or a third AI model (not shown) operating on another external device. When multiple AI models are selected, the orchestration module (467) may communicate with the selected models or devices so that the operation between the selected AI models and the processing of the results thereof can be coordinated between the relevant models or devices.

[0121] According to one embodiment, two or more modules of the generative AI system (400) (e.g., cross-application action module (461) and orchestration module (467)) may be implemented as a single module to maintain the same functionality. Various other variations are also possible.

[0122]

[0123] FIG. 5 schematically illustrates the operation method of a response system according to one embodiment.

[0124] A response system (hereinafter, system) (5) according to one embodiment may include an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, or electronic device (301) of FIG. 3a and FIG. 3b) and an external electronic device (520) (e.g., electronic device (102) of FIG. 1 or electronic device (104)).

[0125] According to one embodiment, the electronic device may be a device such as a mobile terminal (e.g., a smartphone, tablet, or laptop) or a fixed terminal (e.g., a PC (personal computer)).

[0126] According to one embodiment, the electronic device may include at least some of the configurations of the electronic device (201) of FIG. 2 and / or the electronic device (301) of FIG. 3a and FIG. 3b. The electronic device may be implemented in the form of smart glasses, for example, including a wearable electronic device (e.g., the electronic device (201) of FIG. 2), such as virtual reality glasses. The electronic device may also be implemented in the form of a wearable electronic device (e.g., the electronic device (301) of FIG. 3a and FIG. 3b), such as a head-mounted display (HMD), including an augmented reality (AR) device, a virtual reality (VR) device, and / or a mixed reality (MR) device. The electronic device may be configured to easily control user interface (UI) components provided in an AR environment, a VR environment, and / or an MR environment.

[0127] According to one embodiment, the system (5) may be a system for retrieval-augmented generation (RAG).

[0128] Large language models (LLMs), such as the generative AI model (430) of Fig. 4a, are artificial intelligence technologies that support intelligent chatbots and other natural language processing applications, and can generate results for tasks such as responding to user questions, language translation, and sentence completion.

[0129] Search augmentation generation is a process that allows the LLM to refer to a trusted knowledge base outside of its training data source before generating a response. In the system (5), relevant data can be retrieved from external data sources existing in various forms, such as an API (application program interface), a database, or a repository, based on a user's query. The LLM can generate a final response using the retrieved data along with the user's query as input.

[0130] According to one embodiment, the system (5) may include a storage (50) for retrieving related data based on a user's query. For example, an electronic device may include a storage (50) based on original data.

[0131] For example, the original data may include an image (e.g., a still image or a video) of the actual space (or physical environment) surrounding the electronic device. The original data may include sound (or acoustic data) surrounding the electronic device. Due to the vast capacity of the continuously received original data, it is difficult to store all of it in the storage (50) or to maintain the data in its original form.

[0132] According to one embodiment, the storage (50) may include a short-term memory level (or a sub-store of the short-term memory level) (510) and a long-term memory level (or a sub-store of the long-term memory level) (530). Hereinafter, the electronic device may perform any processing (e.g., a first processing, a second processing and / or a third processing) on ​​the original data to overcome the capacity limitations of the storage (50) and to build a data pool for generating an effective response to a user's query.

[0133] According to one embodiment, the electronic device may perform a first processing to simplify or compress original data. The electronic device may store the first data (51) generated by performing the first processing on the original data in a short-term memory level (510) (or in a sub-store of the short-term memory level (510)).

[0134] According to one embodiment, the electronic device may perform a second processing on at least a portion of the first data (51) when the capacity of the first data (51) in the short-term memory level (510) reaches a threshold, or to degrade (or degrade) or compress the quality of a portion of the first data (51) that is of low importance. The electronic device may store the second data (53) generated by performing the second processing on the portion of the first data (51) in the long-term memory level (530) (or in a sub-store of the long-term memory level (530).

[0135] According to one embodiment, the long-term memory level (530) may include a plurality of long-term memory levels (e.g., a first long-term memory level (531) and a second long-term memory level (532)) according to the degree of deterioration.

[0136] For example, the first long-term memory level (531) may include the first data portion (55) of the second data (53). If the first long-term memory level (531) corresponds to the long-term memory level with the lowest degree of degradation among a plurality of long-term memory levels, the electronic device may store the portion generated as a result of performing a second processing on any part of the first data (51) of the short-term memory level (510) as the first data portion (55) of the first long-term memory level (531).

[0137] According to one embodiment, the electronic device may perform a third processing on at least a portion of the first data portion (55) when the capacity of the first data portion (55) of the first long-term memory level (531) reaches a threshold, or to degrade (or degrade) the quality of a portion of the first data portion (55) that is of low importance or to compress it.

[0138] For example, the second long-term memory level (532) may include a second data portion (57) of the second data (53). If the second long-term memory level (532) corresponds to a long-term memory level with a higher degree of degradation than the first long-term memory level (531), the electronic device may store a portion generated as a result of performing a third processing on any part of the first data portion (55) of the first long-term memory level (531) as the second data portion (57) of the second long-term memory level (532).

[0139] The number of long-term memory levels shown in FIG. 5 is exemplary, for example, the long-term memory level (530) may include a medium-long-term memory level and an ultra-long-term memory level. For example, the long-term memory level (530) may include a plurality of long-term memory levels according to the degree of degradation (e.g., degradation level 1 to degradation level 10), but this is not limited to the present disclosure.

[0140] A method for creating and managing a storage (50) is described in detail with reference to FIG. 11.

[0141] According to one embodiment, an electronic device may create sessions to track and manage consecutive or similar queries by a user. The electronic device may also manage multiple sessions individually.

[0142] According to one embodiment, the electronic device may maintain a session for a specified period of time. The electronic device may terminate a session after the specified period has elapsed. The electronic device may also terminate a session that has been deleted directly by the user.

[0143] According to one embodiment, the electronic device can generate session memory data corresponding to each session. The session memory data may represent data used to generate a response to a user query received within the corresponding session.

[0144] According to one embodiment, the electronic device may determine (or generate) session memory data based on a portion of the first data (51) of the short-term memory level (510) and / or a portion of the second data (53) of the long-term memory level (530) contained in the storage (50). A method for determining session memory data is described in detail with reference to FIG. 8.

[0145] According to one embodiment, the electronic device may maintain a session and session memory data corresponding to that session for a specified period of time. The electronic device may terminate a session after the specified period has elapsed and delete the session memory data corresponding to that session.

[0146] According to one embodiment, when an electronic device receives a user query, it can determine a target session for generating a response to the query. A method for determining a target session is described in detail with reference to FIG. 7.

[0147] According to one embodiment, the electronic device can determine session memory data to be used to generate a response to a user's query during a target session.

[0148] For example, an electronic device may construct new session memory data when a new target session is created in response to a currently received user query. For example, if one or more existing sessions are determined to be the target session in response to a currently received user query, the electronic device may update existing session memory data corresponding to that target session based on the query. In this case, if any part of the data within the session memory data is old or not frequently used, that part may be deleted.

[0149] According to one embodiment, the electronic device can generate a response to a user's query based at least on session memory data. For example, the electronic device can determine a prompt to be entered into the LLM using the user's query and session memory data. The session memory data can be used as a database for a type of search augmentation. For example, to generate a response to a user's query, the session memory data and training data of the LLM and / or data obtained from an external source (e.g., the web or an external database) may be used.

[0150] According to one embodiment, the electronic device may determine the importance of each part of the data (e.g., first data (51), first data part (55) or second data part (57)) based on a timestamp regarding the part and the frequency with which the part is utilized for search augmentation to generate a response for the user.

[0151] The electronic device may perform processing (e.g., a second processing or a third processing) to degrade (or degrade) or compress the quality of parts of data that are old and / or parts of data that are not frequently used for search augmentation. The second processing and the third processing are described in detail with reference to FIGS. 12 to 14. If parts of data that are old or highly degraded are frequently used for search augmentation, the electronic device may perform processing (e.g., a target processing) to restore those parts.

[0152] FIG. 6 is a flowchart of a method for generating a response according to one embodiment.

[0153] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0154] According to one embodiment, the following operations 610 to 670 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions.

[0155] According to one embodiment, the electronic device may be a device such as a mobile terminal (e.g., a smartphone, tablet, or laptop) or a fixed terminal (e.g., a PC (personal computer)).

[0156] According to one embodiment, the electronic device may include at least some of the configurations of the electronic device (201) of FIG. 2 and / or the electronic device (301) of FIG. 3a and FIG. 3b. The electronic device may be implemented in the form of a smart glass, for example, a wearable electronic device (e.g., the electronic device (201) of FIG. 2), such as virtual reality glasses. The electronic device may also be implemented in the form of a wearable electronic device (e.g., the electronic device (301) of FIG. 3a and FIG. 3b), such as a head-mounted display (HMD), such as an augmented reality (AR) device, a virtual reality (VR) device, and / or a mixed reality (MR) device. The electronic device may be configured to easily control user interface (UI) components provided in an AR environment, a VR environment, and / or an MR environment.

[0157] According to one embodiment, the electronic device may be a VST type configured to block external light so that when worn, light emitted from a display reaches the user's eyes, but external light does not reach the user's eyes. According to one embodiment, the electronic device may include an OST type configured to allow external light to reach the user's eyes through glasses when worn.

[0158] The electronic device may include a sensor (e.g., sensor module (176) of FIG. 1, camera module (180), first camera (265a, 265b), second camera (270a, 270b), third camera (245) of FIG. 2, second functional camera (311, 312) of FIG. 3a, first functional camera (315), depth sensor (317), third functional camera (328) of FIG. 3b, and / or fourth functional camera (325, 326)). According to one embodiment, the sensor may convert the measured or detected information into an electrical signal (or sensing data) by measuring or detecting a physical quantity. For example, the sensor may include at least one camera or image sensor for capturing at least one frame of a still image or video of real space (or, physical environment). For example, the sensor may include at least one of a button for touch input, a gesture sensor, a gyroscope, a gyro sensor, a barometric pressure sensor, a magnetic sensor, a magnetometer, an accelerometer, an accelerometer, a grip sensor, a proximity sensor, an RGB sensor, a biophysical sensor, a temperature sensor, a humidity sensor, an illuminance sensor, a UV sensor, an electromyography sensor, an electroencephalography sensor, an infrared sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor, but the present disclosure is not limited thereto.

[0159] According to one embodiment, the sensor can capture a physical environment including an object. For example, the sensor may include at least one of an image sensor, a LiDAR sensor, an RGB-D (red-green-blue depth) sensor, a depth sensor, a ToF (time of flight) sensor, an ultrasonic sensor, a radar sensor, and a stereo camera, but the present disclosure is not limited thereto.

[0160] According to one embodiment, the sensor may generate sensing data. The sensing data may be at least one still image or video of a physical environment. The sensing data may be an image (or actual space image) in which one or more objects included in the physical environment are captured. The sensing data may include depth information. For example, the sensing data may be a color image containing depth information, such as an RGB-D image.

[0161] According to one embodiment, an electronic device can acquire sensing data from a sensor. The electronic device can provide augmented reality content using the sensing data acquired from the sensor. The electronic device can generate augmented reality content (or an image of augmented reality content) by blending a physical environment and a virtual environment based on the sensing data. The augmented reality content may include one or more physical environment objects and / or virtual environment objects, such as user interface elements (e.g., avatars, control elements, interactive elements, or any graphic elements), included in a physical environment captured in real time by the electronic device.

[0162] According to one embodiment, the electronic device may acquire original data using the electronic device’s camera (e.g., sensor module (176) of FIG. 1, camera module (180), first camera (265a, 265b), second camera (270a, 270b), third camera (245) of FIG. 2, second functional camera (311, 312), first functional camera (315), depth sensor (317) of FIG. 3a, third functional camera (328) of FIG. 3b, or fourth functional camera (325, 326)). The original data may include an image (e.g., a still image or a video) of the actual space (or physical environment) surrounding the electronic device.

[0163] According to one embodiment, the electronic device may acquire original data using the electronic device's microphone (e.g., the input module (150) of FIG. 1, the first microphone (250a), the second microphone (250b), or the third microphone (250c) of FIG. 2). The original data may include sound (or acoustic data) around the electronic device.

[0164] In operation 610, the electronic device can generate first data at the short-term memory level by performing a first processing on the original data acquired by the electronic device.

[0165] According to one embodiment, the first processing may include, for each of the images of the original data, at least one of vectorization of an object appearing in the image, generation of object information by layer of the image, or generation of metadata of the image. The first processing is described in detail with reference to FIG. 11.

[0166] In operation 620, the electronic device can generate second data at the long-term memory level by performing second processing on at least a portion of the first data.

[0167] According to one embodiment, the second processing may include, for each of at least a portion of the images of the first data, at least one of resizing the image, merging the image with a portion of the previously generated second data, or summarizing the image. The second processing is described in detail with reference to FIG. 11.

[0168] According to one embodiment, the long-term memory level may include a plurality of long-term memory levels according to the degree of degradation. A method for subdividing and refining the second data according to the plurality of long-term memory levels is described in detail with reference to FIG. 14.

[0169] In operation 630, the electronic device can receive a user's query.

[0170] According to one embodiment, the electronic device may receive a user's query in the form of text data. According to one embodiment, the electronic device may receive a user's query in the form of voice data. For example, the electronic device may receive the user's voice using a microphone and obtain the user's query by analyzing the received voice. The electronic device may include a neural network-based model that obtains the user's query based on voice.

[0171] For example, an electronic device can receive a user's query requesting a response using media such as photos, such as, "Show me the picture of the puppy taken last week," or "What was the blue dress I wore on vacation last summer?"

[0172] According to one embodiment, an electronic device may create a session to track and manage consecutive or similar queries by a user. According to one embodiment, the electronic device may maintain the session for a specified period of time. The electronic device may terminate the session after the specified period of time has elapsed.

[0173] According to one embodiment, the electronic device can determine context information of a user's query. For example, the context information of the user's query may include the subject or keyword of the query. For example, the context information of the user's query may include an embedding representation of the query in the form of text data (e.g., vector embedding).

[0174] According to one embodiment, the electronic device may store context information of a session while maintaining the session. The context information of the session may include context information of at least one user query received within the session. For example, if multiple queries are received within the session, the electronic device may generate and store a topic or keyword of the session based on the queries.

[0175] In operation 640, the electronic device can determine a target session to generate a response to a query based on context information of the user's query.

[0176] The electronic device may determine a specific session that has already been created as a target session for generating a response to a query received in operation 640, or create a new target session, by comparing the context information of a user's query with the context information of a session that has already been created (or maintained). A method for determining a target session for generating a response to a user's query is described in detail with reference to FIG. 7.

[0177] In operation 650, the electronic device can determine session memory data corresponding to a target session that includes at least one of a part of the first data or a part of the second data based on context information of the user's query.

[0178] According to one embodiment, the electronic device may generate session memory data corresponding to each session. The session memory data may represent data used to generate a response to a user query received within the corresponding session. According to one embodiment, the electronic device may maintain a session and the session memory data corresponding to that session for a specified period of time. The electronic device may terminate a session after the specified period has elapsed and delete the session memory data corresponding to that session.

[0179] An electronic device may determine (or generate) session memory data using a portion of the first data at the short-term memory level and the second data at the long-term memory level. For example, based on contextual information of a user's query, the electronic device may determine at least one memory level among the short-term memory level and / or long-term memory level to be used for generating session memory data according to a specific time specified or implied by the user's query. For example, based on contextual information of a user's query, the electronic device may determine a specific part (or data part) similar to the user's query from the first data at the short-term memory level and / or the second data at the long-term memory level to be used for generating session memory data.

[0180] A method for determining session memory data is explained in detail with reference to Fig. 7.

[0181] In operation 660, the electronic device can generate a response to a user's query based at least on session memory data.

[0182] According to one embodiment, an electronic device can obtain a response output by a trained model by inputting a portion of first data, determined by a user's query and session memory data, into the trained model. For example, the electronic device can obtain a response using a trained model such as the generative AI model (430) of FIG. 4a.

[0183] According to one embodiment, an electronic device may determine (or generate) a prompt based on a user's query and session memory data. The electronic device may determine a prompt that includes the user's query, session memory data, and instructions (or commands) to generate a response to the user's query using the user's query and session memory data. The electronic device may obtain a response to the user's query output by a trained model by inputting the prompt into a trained model.

[0184] In operation 670, the electronic device can output a response to the user's query.

[0185] The response to a user's query may be in the form of visual data (e.g., images or text) and / or voice data.

[0186] For example, the electronic device may display a response in the form of an image through a display (e.g., the display module (160) of FIG. 1, the display (205, 210) of FIG. 2, or the display (321) of FIG. 3b). For example, the electronic device may display a response in the form of text through a display. For example, the electronic device may play a response in the form of voice data.

[0187] FIG. 7 is a flowchart of a method for determining a target session according to one embodiment.

[0188] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0189] According to one embodiment, the following operations 710 and 720 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions.

[0190] As described with reference to FIG. 6, the electronic device can receive a user's query. According to one embodiment, the electronic device can receive a user's query in the form of text data. According to one embodiment, the electronic device can receive a user's query in the form of voice data.

[0191] According to one embodiment, the operation 640 for determining a target session for generating a response to a query of FIG. 6 may include operations 710 to 730.

[0192] According to one embodiment, an electronic device may create a session to track and manage consecutive or similar queries by a user. According to one embodiment, the electronic device may maintain the session for a specified period of time. The electronic device may terminate the session after the specified period of time has elapsed.

[0193] According to one embodiment, the electronic device can determine context information of a user's query. The context information of the user's query may include the subject or keyword of the query.

[0194] According to one embodiment, the electronic device may store context information of a session while maintaining the session. The context information of the session may include context information of at least one user query received within the session. For example, if multiple queries are received within the session, the electronic device may generate and store a topic or keyword of the session based on the queries.

[0195] In operation 710, the electronic device can determine whether there exists a session among one or more sessions created for a user that corresponds to the context information of the query.

[0196] When a user's query is received, the electronic device may compare the context information of the query with the context information of one or more sessions that have already been created (or are being maintained) for the user. For example, the electronic device may determine the similarity (e.g., cosine similarity) or distance (e.g., Euclidean distance) between the context information of the query and the context information of one or more sessions created for the user.

[0197] In operation 720, if there is a session among one or more sessions created for a user that corresponds to the context information of a query, the electronic device may determine the session corresponding to the context information of the query as the target session.

[0198] For example, the electronic device may determine a session as a target session if, among one or more sessions generated for a user, there exists a session in which the similarity to the context information of a query is greater than or equal to a threshold, or the distance to the context information of a query is less than or equal to a threshold. If, among one or more sessions generated for a user, there exist multiple sessions in which the similarity to the context information of a query is greater than or equal to a threshold, or the distance to the context information of a query is less than or equal to a threshold, the electronic device may determine the session with the greatest similarity to the query or the smallest distance among the multiple sessions as the target session.

[0199] Accordingly, the electronic device can generate a response to a user's query using previous queries similar to the user's query and / or data used to generate responses to previous queries (e.g., session memory data) that are maintained with respect to the target session.

[0200] In operation 730, if there is no session created for the user, or if there is no session among one or more sessions created for the user that corresponds to the context information of the query, the electronic device may create a target session to create a response to the query.

[0201] For example, if there is no session among one or more sessions created for a user in which the similarity to the context information of the query is above (or exceeds) a threshold, or the distance to the context information of the query is below (or less than) a threshold, the electronic device may create a new target session to generate a response to a query. In this case, the electronic device may construct data (e.g., session memory data) to be used to generate a response to the user's query during the target session.

[0202] FIG. 8 is a flowchart of a method for determining session memory data according to one embodiment.

[0203] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0204] According to one embodiment, the following operations 810 to 830 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions.

[0205] As described with reference to FIG. 6, an electronic device can receive a user's query. According to one embodiment, the electronic device can receive a user's query in the form of text data. According to one embodiment, the electronic device can receive a user's query in the form of voice data. The electronic device can determine a target session for generating a response to the query based on contextual information of the user's query.

[0206] According to one embodiment, the operation 650 for determining session memory data corresponding to the target session of FIG. 6 may include operations 810 to 830.

[0207] In operation 810, the electronic device can determine at least one memory level corresponding to the user's query among short-term memory levels and long-term memory levels based on context information of the user's query.

[0208] According to one embodiment, an electronic device can identify time information corresponding to a user's query based on context information of the user's query. That is, the electronic device can identify time information specified or implied by the user's query based on context information of the user's query. The electronic device can determine at least one memory level among a short-term memory level and a long-term memory level that corresponds to the time information identified based on context information of the user's query.

[0209] For example, when an electronic device receives a user query such as "Show me the outfit I wore yesterday" or "Show me the street performance I saw on my trip to Paris two months ago," it can identify time information such as "yesterday" or "two months ago." The electronic device can determine a short-term memory level as at least one memory level corresponding to time information such as "yesterday" or "two months ago."

[0210] For example, when an electronic device receives a query such as, "What was the vibe of the blue dress I wore in Jeju 10 years ago?", it can identify time information such as "10 years ago." The electronic device can determine a long-term memory level as at least one memory level corresponding to time information such as "10 years ago."

[0211] For example, when an electronic device receives a query such as "Show me a picture of the beach from when I went on vacation to region A every summer," it can identify time information such as "every year." The electronic device can determine a short-term memory level and a long-term memory level as at least one memory level corresponding to time information such as "every year."

[0212] According to one embodiment, the electronic device may determine at least one memory level corresponding to time information identified based on context information of a user's query among a short-term memory level and a long-term memory level according to a predetermined time condition. For example, if the time information identified based on context information of the user's query indicates a point in time that is less than or equal to a threshold time, the electronic device may determine a short-term memory level as the memory level corresponding to the user's query. For example, if the time information identified based on context information of the user's query indicates a point in time that is greater than or equal to a threshold time, the electronic device may determine a long-term memory level as the memory level corresponding to the user's query.

[0213] According to one embodiment, if the electronic device cannot identify time information corresponding to the user's query based on context information of the user's query, it can determine a short-term memory level as at least one memory level corresponding to the user's query.

[0214] For example, when an electronic device receives a query such as 'Show me a picture of my dog,' it may determine that it cannot identify time information corresponding to the query. If specific time information is not specified or implied by the user's query, the electronic device may determine a short-term memory level as the memory level corresponding to the user's query.

[0215] The electronic device can determine whether a part (or a data part) similar to a user's query (or contextual information of the query) exists in the first data of the short-term memory level. If the electronic device does not have a part similar to the user's query in the first data of the short-term memory level, it can determine a long-term memory level as at least one memory level corresponding to the user's query. The electronic device can determine whether a part similar to the user's query exists in the second data of the long-term memory level. By considering the first data of the short-term memory level first and considering the second data of the long-term memory level second in advance, the electronic device can determine session memory data using more accurate and clear data corresponding to the user's query.

[0216] In operation 820, the electronic device can determine target data containing at least one of a part of the first data or a part of the second data based on at least one memory level.

[0217] According to one embodiment, when a short-term memory level is determined as at least one memory level corresponding to a user's query, the electronic device can determine a portion of the first data of the short-term memory level as target data based on context information of the user's query.

[0218] According to one embodiment, when a long-term memory level is determined as at least one memory level corresponding to a user's query, the electronic device can identify a portion of the second data of the long-term memory level based on context information of the user's query.

[0219] According to one embodiment, the electronic device can determine whether there exists an object determined (or identified) based on the user's query (or context information) in at least one memory level of data (e.g., first data and / or second data) corresponding to the user's query.

[0220] The electronic device has determined a short-term memory level as at least one memory level corresponding to a user's query, and if an object determined based on the user's query exists in the first data of the short-term memory level, the part of the first data corresponding to the object can be determined as target data.

[0221] The electronic device has determined a long-term memory level as at least one memory level corresponding to a user's query, and if an object determined based on the user's query exists in the second data of the long-term memory level, the part of the second data corresponding to the object can be determined as target data.

[0222] Even when the long-term memory level is determined as the memory level corresponding to the user's query according to a specific time specified or implied by the user's query, the electronic device may additionally consider first data of a short-term memory level satisfying a defined condition regarding the user's query so that session memory data can be determined using more accurate and clear data of the short-term memory level. The electronic device may additionally identify a portion of the first data of the short-term memory level based on contextual information of the user's query.

[0223] According to one embodiment, the electronic device may determine that a long-term memory level is determined as at least one memory level corresponding to a user's query, but that an object determined based on the user's query does not exist in the second data of the long-term memory level. If the electronic device determines that a long-term memory level is determined as at least one memory level corresponding to a user's query, but that an object determined based on the user's query does not exist in the second data of the long-term memory level, the electronic device may determine whether the corresponding object exists in the first data of the short-term memory level. If the electronic device determines that the corresponding object exists in the first data of the short-term memory level, it may determine the portion of the first data corresponding to the object as target data.

[0224] According to one embodiment, when a long-term memory level is determined as at least one memory level corresponding to a user's query, the electronic device can determine whether an object determined based on the user's query exists in each of the first data and the second data.

[0225] According to one embodiment, the electronic device determines a long-term memory level as at least one memory level corresponding to a user's query, and if an object determined based on the user's query exists only in the second data of the long-term memory level and not in the first data of the short-term memory level, the part of the second data corresponding to the object can be determined as target data.

[0226] According to one embodiment, the electronic device has determined a long-term memory level as at least one memory level corresponding to a user's query, and if an object determined based on the user's query exists in a first data of a short-term memory level and a second data of a long-term memory level, respectively, a part of the first data and a part of the second data corresponding to the object can be determined as target data.

[0227] For example, as described below with reference to FIG. 14, the electronic device may transfer a specific part of the existing second data to the first data at the short-term memory level when that part is frequently used for search augmentation (or when it is of high importance). For example, the electronic device may store an object determined based on the user's query as part of the data at the long-term and short-term memory levels when the same or similar original data is repeatedly acquired, such as when the user wore the same clothes five years ago and one week ago. Thus, the electronic device can effectively provide a response using the data at the short-term memory level even when the long-term memory level is determined as at least one memory level corresponding to the user's query.

[0228] According to one embodiment, the electronic device determines a long-term memory level as at least one memory level corresponding to a user's query, and if an object determined based on the user's query exists in the first data of the short-term memory level and the second data of the long-term memory level, respectively, only the portion of the first data corresponding to the object can be determined as target data. Accordingly, the electronic device can provide a response using only relatively clearer data.

[0229] According to one embodiment, when the electronic device determines a short-term memory level and a long-term memory level as at least one memory level corresponding to a user's query, it can determine a portion of the first data of the short-term memory level and a portion of the second data of the long-term memory level as target data based on context information of the user's query.

[0230] The method for determining target data is explained in detail with reference to Fig. 9.

[0231] In operation 830, the electronic device can determine session memory data corresponding to the target session based on the target data.

[0232] According to one embodiment, when a portion of first data at a short-term memory level is determined as target data, the electronic device can determine the portion of first data determined as target data as session memory data.

[0233] According to one embodiment, when a portion of second data at a long-term memory level is determined as target data, the electronic device can determine the portion of second data determined as target data as session memory data.

[0234] According to one embodiment, when a portion of first data and a portion of second data are determined as target data, the electronic device can determine session memory data corresponding to a target session by performing a target processing to restore a portion of second data based on a portion of first data.

[0235] Target processing may include at least one of upscaling, inpainting, or outpainting. When a portion of the first data and a portion of the second data are determined to be target data, the electronic device may restore a portion of the second data that is degraded compared to the portion of the first data using the portion of the first data. Session memory data may include the portion of the second data and the portion of the first data restored as a result of target processing.

[0236] When target processing is performed on a portion of second data satisfying a predetermined condition based on a portion of first data satisfying a predetermined condition regarding a query, the electronic device can determine the result of the target processing as the first data at the short-term memory level. That is, the electronic device can include the restored portion of second data in the first data at the short-term memory level.

[0237] FIG. 9 is a flowchart of a method for determining session memory data according to one embodiment.

[0238] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0239] According to one embodiment, the following operations 910 to 990 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions.

[0240] As described with reference to FIG. 6, an electronic device can receive a user's query. According to one embodiment, the electronic device can receive a user's query in the form of text data. According to one embodiment, the electronic device can receive a user's query in the form of voice data. The electronic device can determine a target session for generating a response to the query based on contextual information of the user's query.

[0241] As described with reference to FIG. 8, the electronic device can determine at least one memory level corresponding to the user's query among short-term memory levels and long-term memory levels based on context information of the user's query.

[0242] According to one embodiment, the operation 820 for determining target data including at least one part of the first data or at least one part of the second data based on at least one memory level of FIG. 8 may include operations 920 to 990.

[0243] In operation 910, the electronic device can determine at least one memory level corresponding to the user's query among short-term memory levels and long-term memory levels based on context information of the user's query.

[0244] As described with reference to FIG. 8, according to one embodiment, an electronic device can identify time information corresponding to a user's query based on context information of the user's query. The electronic device can determine at least one memory level among a short-term memory level and a long-term memory level that corresponds to the time information identified based on context information of the user's query. According to one embodiment, the electronic device can determine at least one memory level among a short-term memory level and a long-term memory level that corresponds to the time information identified based on context information of the user's query according to a predetermined time condition. According to one embodiment, if the electronic device cannot identify time information corresponding to the user's query based on context information of the user's query, it can determine a short-term memory level as at least one memory level corresponding to the user's query.

[0245] Hereinafter, the electronic device may determine as target data any part of at least one memory level of data (e.g., first data and / or second data) corresponding to the user's query that satisfies a defined condition regarding the user's query.

[0246] According to one embodiment, the defined condition may include that the similarity (e.g., cosine similarity) between the context information of a user's query and a portion of data at any memory level (e.g., first data or second data) is greater than or equal to a threshold. According to one embodiment, the defined condition may include that the distance (e.g., Euclidean distance) between the context information of a user's query and a portion of data at any memory level (e.g., first data or second data) is less than or equal to a threshold.

[0247] According to one embodiment, the defined condition may include the existence of an object determined (or identified) based on the user's query (or context information) in at least one memory level of data (e.g., first data and / or second data) corresponding to the user's query.

[0248] An electronic device can identify the most important object (or object name) in a user's query, the purpose of the question itself, or the object most relevant to the purpose of the question. For example, if the electronic device receives a user query such as "Do you remember the red blouse I wore in Gangnam two months ago?", it can identify "red blouse."

[0249] An electronic device can determine whether an object determined (or identified) based on a user's query (or context information) exists in at least one memory level of data (e.g., first data and / or second data) corresponding to a user's query. For example, the electronic device can determine whether any part (or image) exists in at least one memory level of data such that the similarity between the color of a shape included in a corresponding vector image format and / or text describing each part and the object determined based on the user's query is greater than (or exceeds) a threshold.

[0250] According to one embodiment, after the operation 910 of determining at least one memory level corresponding to a user's query, the electronic device can determine whether there exists any part of the data (e.g., first data and / or second data) of the at least one memory level corresponding to the user's query that satisfies a defined condition regarding the user's query.

[0251] In operation 920, if the electronic device determines a short-term memory level as at least one memory level corresponding to the user's query based on context information of the user's query, it can determine a portion of the first data that satisfies a defined condition regarding the user's query as target data.

[0252] When a short-term memory level is determined as at least one memory level corresponding to a user's query, the electronic device can compare each part of the first data of the short-term memory level with the context information of the query.

[0253] The electronic device determines a short-term memory level as at least one memory level corresponding to the user's query based on contextual information of the user's query, and if there is a part of the first data that satisfies a defined condition regarding the user's query, that part can be determined as target data.

[0254] According to one embodiment, if there exists a part of the first data where the similarity with the context information of the query is greater than or equal to a threshold (or exceeds) or the distance from the context information of the query is less than or equal to a threshold, the electronic device may determine that part as target data. According to one embodiment, if there exists an object determined based on a user's query in the first data, the electronic device may determine a part (or image) of the first data corresponding to said object as target data.

[0255] In the operation 830 of determining session memory data corresponding to a target session based on the target data of FIG. 8, a portion of the first data determined as the target data may be determined as session memory data.

[0256] In an electronic device, a short-term memory level is determined as at least one memory level corresponding to a user's query, but if there is no part of the first data that satisfies a defined condition regarding the user's query, the context information of the query can be compared with each part of the second data of the long-term memory level.

[0257] If there is no part of first data satisfying a defined condition regarding a user's query, the electronic device can determine whether there is a part of second data satisfying a defined condition regarding a user's query. If there is a part of second data satisfying a defined condition regarding a user's query, the electronic device can determine the part of second data as target data.

[0258] According to one embodiment, if there exists a part of the second data where the similarity with the context information of the query is greater than or equal to a threshold (or exceeds) or the distance from the context information of the query is less than or equal to a threshold, the electronic device may determine that part as target data. According to one embodiment, if there exists an object determined based on a user's query in the second data, the electronic device may determine a part (or image) of the second data corresponding to said object as target data.

[0259] In operation 930, if a long-term memory level is determined as at least one memory level corresponding to the user's query based on context information of the user's query, the electronic device can identify a portion of second data that satisfies a defined condition regarding the user's query.

[0260] When a long-term memory level is determined as at least one memory level corresponding to a user's query, the electronic device can compare the context information of the query with each part of the second data of the long-term memory level.

[0261] The electronic device has determined a long-term memory level as at least one memory level corresponding to the user's query based on context information of the user's query, and if there is a part in the second data that satisfies a defined condition regarding the user's query, the corresponding part can be identified.

[0262] According to one embodiment, the electronic device can identify a portion of the second data if there exists a portion in which the similarity with the context information of the query is greater than or equal to a threshold (or exceeds) or the distance from the context information of the query is less than or equal to a threshold. According to one embodiment, if an object determined based on a user's query exists in the second data, the electronic device can identify a portion (or image) of the second data corresponding to the object.

[0263] According to one embodiment, the electronic device may proceed to operation 950 after operation 930, which identifies a portion of second data that satisfies a defined condition regarding a user's query.

[0264] In operation 950, the electronic device can determine whether there exists a portion of the first data of the short-term memory level that satisfies a defined condition regarding the user's query.

[0265] The electronic device may additionally consider first data of a short-term memory level satisfying a defined condition regarding the user's query so that session memory data can be determined using more accurate and clear data of a short-term memory level even when the long-term memory level is determined as the memory level corresponding to the user's query according to a specific time specified or implied by the user's query.

[0266] In operation 960, if there exists a part of first data that satisfies a defined condition regarding a user's query, the electronic device may determine the part of first data and the part of second data identified in operation 930 as target data.

[0267] In operation 970, if there is no part of the first data satisfying a defined condition regarding the user's query, the electronic device may determine the part of the second data identified in operation 930 as the target data.

[0268] According to one embodiment, if a long-term memory level is determined as at least one memory level corresponding to a user query based on context information of the user query, the electronic device can identify a portion of second data containing an object determined based on the user query. The electronic device can determine whether the object determined based on the user query also exists in the first data. If the portion of first data containing the object determined based on the user query exists, the electronic device can determine the portion of first data as target data. Accordingly, the electronic device can provide a response using only relatively clearer data.

[0269] According to one embodiment, the electronic device may proceed to operation 940 after operation 930, which identifies a portion of second data that satisfies a defined condition regarding a user's query.

[0270] In operation 940, the electronic device can determine whether the degree of degradation of a portion of the second data satisfying a defined condition regarding a user's query is greater than (or greater than) a threshold.

[0271] According to one embodiment, the electronic device can determine whether a portion of second data satisfying a defined condition regarding a user's query corresponds to a defined long-term memory level among a plurality of long-term memory levels where the degree of degradation is greater than (or greater than) a threshold.

[0272] In operation 980, if the electronic device obtains external data regarding a user's query from an external source of the electronic device when the portion of the second data identified as satisfying a defined condition regarding the query corresponds to a defined long-term memory level among a plurality of long-term memory levels where the degree of degradation is greater than (or greater than) a threshold value.

[0273] External data regarding a user's query may include data in which the similarity to the user's query (or contextual information of the user's query) retrieved from an external source is greater than or equal to a threshold, or the distance from the query is less than or equal to a threshold. For example, an electronic device may obtain external data regarding a user's query through a web search. For example, an electronic device may obtain external data regarding a user's query from a database for search augmentation other than the storage of the present disclosure (e.g., storage (50) of FIG. 5).

[0274] In operation 990, the electronic device can determine a portion of second data that satisfies defined conditions regarding external data and queries as target data. Session memory data corresponding to the target session can be determined by performing target processing to restore a portion of second data based on external data.

[0275] Target processing may include at least one of upscaling, inpainting, or outpainting. When the external data and a portion of the second data are determined to be the target data, the electronic device may restore a relatively degraded portion of the second data using the external data. Session memory data may include the restored portion of the second data and the external data.

[0276] Again in operation 940, the electronic device can determine whether the degree of degradation of the portion of the second data satisfying the defined conditions regarding the query is greater than a threshold. If the degree of degradation of the portion of the second data is below (or less than) the threshold, the electronic device may proceed to operation 950. Descriptions of operations 950 through 970 that overlap with the previously described content are omitted.

[0277] Again in operation 910, the electronic device can determine at least one memory level corresponding to the user's query. After operation 910, the electronic device can determine whether there exists any part in the data of at least one memory level corresponding to the user's query (e.g., first data and / or second data) that satisfies a defined condition regarding the user's query.

[0278] According to one embodiment, when a long-term memory level is determined as at least one memory level corresponding to a user's query based on context information of the user's query, the electronic device can determine whether there exists a part of second data satisfying a predetermined condition regarding the user's query.

[0279] If there is no part of second data satisfying a defined condition regarding the user's query, the electronic device can determine whether there is a part of first data satisfying a defined condition regarding the user's query. If there is a part of first data satisfying a defined condition regarding the user's query, the electronic device can determine the part of first data as target data.

[0280] FIGS. 10a and FIGS. 10b are drawings illustrating a method of providing a response according to one example, respectively.

[0281] As described with reference to FIGS. 6 through 9, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b) may receive a user query. According to one embodiment, the electronic device may receive a user query in the form of text data. According to one embodiment, the electronic device may receive a user query in the form of voice data. The electronic device may determine a target session for generating a response to the query based on contextual information of the user query. Based on contextual information of the user query, the electronic device may determine session memory data corresponding to a target session that includes at least one of a part of first data or a part of second data.

[0282] According to one embodiment, the electronic device can determine context information of a user's query. For example, the context information of the user's query may include the subject or keyword of the query. For example, the context information of the user's query may include an embedding representation of the query in the form of text data (e.g., vector embedding).

[0283] According to one embodiment, the electronic device can identify time information corresponding to a user's query based on context information of the user's query. The electronic device can determine at least one memory level among a short-term memory level and a long-term memory level that corresponds to the time information identified based on context information of the user's query.

[0284] Referring to FIG. 10a, for example, when an electronic device receives a user query such as "Do you remember the red blouse I wore in Gangnam two months ago?", it can identify time information such as "two months ago." The electronic device can determine a short-term memory level as at least one memory level corresponding to time information such as "two months ago."

[0285] When a short-term memory level is determined as at least one memory level corresponding to a user's query based on context information of the user's query, the electronic device can determine a portion of first data satisfying a defined condition regarding the user's query as target data.

[0286] For example, the electronic device can compare keywords such as 'red', 'blouse', and 'Gangnam' as contextual information of a user's query with each part of the first data at the short-term memory level (e.g., an image). The electronic device can compare time information such as 'two months ago' with each part of the first data at the short-term memory level. Specifically, the electronic device can compare keywords such as 'red' and 'blouse' as contextual information of a user's query with the color of a shape included in the vector image format of each part of the first data and / or text describing each part. The electronic device can compare time information such as 'two months ago' with the timestamp of each part of the first data. The electronic device can compare keywords indicating a location, such as 'Gangnam', with the location where each part of the first data was obtained.

[0287] The electronic device may determine a portion of the first data as target data if there exists a portion in which the similarity with the context information of the query (and / or time information based on the context information) is greater than or equal to a threshold, or the distance from the context information of the query is less than or equal to a threshold. The electronic device may determine the portion of the first data determined as target data as session memory data. The electronic device may generate a response to a user's query based at least on the session memory data.

[0288] According to one embodiment, an electronic device can obtain a response output by a trained model by inputting a portion of first data, determined by a user's query and session memory data, into the trained model. For example, the electronic device can obtain a response using a trained model such as the generative AI model (430) of FIG. 4a.

[0289] According to one embodiment, the electronic device may determine (or generate) a prompt based on a user's query and session memory data. For example, in first data at the short-term memory level, a part (or image) of the user wearing a red blouse may be determined as session memory data. The electronic device may determine a prompt containing instructions (or commands) to generate a response to the user's query using the user's query, a part of the first data, and the user's query and the part of the first data. The electronic device may obtain a response to the user's query output by a trained model by inputting the prompt into a trained model.

[0290] The response to a user's query may be in the form of visual data (e.g., image or text) and / or voice data. For example, an electronic device may output a response (1001) in the form of an image. According to one embodiment, the response (1001) in the form of an image may further include text such as 'two months ago' or 'in Gangnam' in a portion of the first data (e.g., image) at the short-term memory level. For example, the electronic device may output a response (1003) in the form of text or voice data.

[0291] Referring to FIG. 10b, for example, when an electronic device receives a user query such as, "What kind of vibe was the blue dress I wore in Jeju 10 years ago?", it can identify time information such as "10 years ago." The electronic device can determine a long-term memory level as at least one memory level corresponding to time information such as "10 years ago."

[0292] When a long-term memory level is determined as at least one memory level corresponding to the user's query based on context information of the user's query, the electronic device can identify a portion of second data satisfying a predetermined condition regarding the user's query. When a long-term memory level is determined as at least one memory level corresponding to the user's query, the electronic device can determine whether a portion of first data in a short-term memory level satisfying a predetermined condition regarding the user's query exists.

[0293] If there exists a part of first data that satisfies a defined condition regarding a user's query, the electronic device can determine the part of first data and the part of second data as target data.

[0294] If there is no part of the first data satisfying a defined condition regarding the user's query, the electronic device may determine only the part of the second data as the target data.

[0295] For example, the electronic device can compare keywords such as 'blue', 'one-piece', and 'Jeju' as contextual information of a user's query with each part of second data at the long-term memory level (e.g., an image). The electronic device can compare time information such as '10 years ago' with each part of second data at the long-term memory level. The electronic device can compare keywords such as 'blue', 'one-piece', and 'Jeju' as contextual information of a user's query with each part of first data at the short-term memory level (e.g., an image).

[0296] The electronic device can identify a corresponding part of the second data if there exists a part in which the similarity with the context information of the query (and / or time information based on the context information) in the second data is greater than (or exceeds) a threshold, or the distance from the context information of the query is less than (or less than) a threshold.

[0297] The electronic device can determine the corresponding part of the first data and the part of the second data as target data if there exists a part in the first data where the similarity with the context information of the query (and / or time information based on the context information) is greater than or equal to a threshold, or where the distance from the context information of the query is less than or equal to a threshold.

[0298] The electronic device can determine only a portion of the second data as target data if there is no portion in the first data where the similarity with the context information of the query is greater than (or exceeds) a threshold, or where the distance from the context information of the query is less than (or less than) a threshold.

[0299] The electronic device can determine session memory data based on target data. The electronic device can generate a response to a user's query based at least on the session memory data.

[0300] According to one embodiment, an electronic device may determine (or generate) a prompt based on a user's query and session memory data. For example, if a part (or image) containing blue clothing is identified in second data at the long-term memory level, and no part similar to the contextual information of the user's query exists in first data at the short-term memory level, a part of the second data may be determined as session memory data. The electronic device may determine a prompt containing instructions (or commands) to generate a response to the user's query using the user's query, the part of the second data, and the user's query and the part of the second data. The electronic device may obtain a response to the user's query output by a trained model by inputting the prompt into a trained model.

[0301] The response to the user's query may be in the form of visual data (e.g., image or text) and / or voice data. For example, the electronic device may output a response (1005) in the form of an image. For example, the electronic device may output a response (1007) in the form of text or voice data.

[0302] Referring to FIGS. 10a and 10b, for example, a response (1001, 1003) generated based on a portion of the first data at the short-term memory level and a response (1005, 1007) generated based on a portion of the second data at the long-term memory level may differ in terms of accuracy and specificity. In response (1001), detailed patterns and shapes of a blouse appear, whereas in response (1005), only the silhouette of a dress may appear. In response (1003), a detailed description of the blouse appears, whereas in response (1007), a relatively simple description may appear.

[0303] As described above, when a long-term memory level is determined as at least one memory level corresponding to a user's query, the electronic device can identify a portion of second data that satisfies a defined condition regarding the user's query. The electronic device can determine whether there exists a portion of first data in a short-term memory level that satisfies a defined condition regarding the user's query.

[0304] According to one embodiment, if there is no part in the first data where the similarity with the context information of the query is greater than (or exceeds) a threshold, or where the distance from the context information of the query is less than (or less than) a threshold, the electronic device may obtain external data regarding the user's query from an external source of the electronic device.

[0305] For example, an electronic device may obtain external data regarding a user's query through a web search. For example, the electronic device may obtain external data regarding a user's query from a database for search augmentation other than the repository of the present disclosure (e.g., the repository (50) of FIG. 5).

[0306] The electronic device may determine external data and a portion of second data as target data. The electronic device may determine session memory data corresponding to a target session by performing target processing to restore a portion of second data based on external data. For example, the electronic device may perform inpainting processing on a portion of second data to infuse features such as the color, material, or shape of a blue dress based on external data, which is an image of a blue dress. The session memory data may include the restored portion of second data and external data. According to one embodiment, the electronic device may generate a response to a user's query based on the user's query and session memory data.

[0307] FIG. 11 is a flowchart of a method for generating and managing data at the short-term and long-term memory level according to one embodiment.

[0308] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0309] According to one embodiment, the following operations 1110 to 1130 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions.

[0310] According to one embodiment, the electronic device may be a device such as a mobile terminal (e.g., a smartphone, tablet, or laptop) or a fixed terminal (e.g., a PC (personal computer)).

[0311] According to one embodiment, the electronic device may include at least some of the configurations of the electronic device (201) of FIG. 2 and / or the electronic device (301) of FIG. 3a and FIG. 3b. The electronic device may be implemented in the form of a smart glass, for example, a wearable electronic device (e.g., the electronic device (201) of FIG. 2), such as virtual reality glasses. The electronic device may also be implemented in the form of a wearable electronic device (e.g., the electronic device (301) of FIG. 3a and FIG. 3b), such as a head-mounted display (HMD), such as an augmented reality (AR) device, a virtual reality (VR) device, and / or a mixed reality (MR) device. The electronic device may be configured to easily control user interface (UI) components provided in an AR environment, a VR environment, and / or an MR environment.

[0312] According to one embodiment, the electronic device may be a VST type configured to block external light so that when worn, light emitted from a display reaches the user's eyes, but external light does not reach the user's eyes. According to one embodiment, the electronic device may include an OST type configured to allow external light to reach the user's eyes through glasses when worn.

[0313] The electronic device may include a sensor (e.g., sensor module (176) of FIG. 1, camera module (180), first camera (265a, 265b), second camera (270a, 270b), third camera (245) of FIG. 2, second functional camera (311, 312) of FIG. 3a, first functional camera (315), depth sensor (317), third functional camera (328) of FIG. 3b, and / or fourth functional camera (325, 326)). According to one embodiment, the sensor may convert the measured or detected information into an electrical signal (or sensing data) by measuring or detecting a physical quantity. For example, the sensor may include at least one camera or image sensor for capturing at least one frame of a still image or video of real space (or, physical environment). For example, the sensor may include at least one of a button for touch input, a gesture sensor, a gyroscope, a gyro sensor, a barometric pressure sensor, a magnetic sensor, a magnetometer, an accelerometer, an accelerometer, a grip sensor, a proximity sensor, an RGB sensor, a biophysical sensor, a temperature sensor, a humidity sensor, an illuminance sensor, a UV sensor, an electromyography sensor, an electroencephalography sensor, an infrared sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor, but the present disclosure is not limited thereto.

[0314] According to one embodiment, the sensor can capture a physical environment including an object. For example, the sensor may include at least one of an image sensor, a LiDAR sensor, an RGB-D (red-green-blue depth) sensor, a depth sensor, a ToF (time of flight) sensor, an ultrasonic sensor, a radar sensor, and a stereo camera, but the present disclosure is not limited thereto.

[0315] According to one embodiment, the sensor may generate sensing data. The sensing data may be at least one still image or video of a physical environment. The sensing data may be an image (or actual space image) in which one or more objects included in the physical environment are captured. The sensing data may include depth information. For example, the sensing data may be a color image containing depth information, such as an RGB-D image.

[0316] According to one embodiment, an electronic device can acquire sensing data from a sensor. The electronic device can provide augmented reality content using the sensing data acquired from the sensor. The electronic device can generate augmented reality content (or an image of augmented reality content) by blending a physical environment and a virtual environment based on the sensing data. The augmented reality content may include one or more physical environment objects and / or virtual environment objects, such as user interface elements (e.g., avatars, control elements, interactive elements, or any graphic elements), included in a physical environment captured in real time by the electronic device.

[0317] In operation 1110, the electronic device can acquire the original data.

[0318] According to one embodiment, the electronic device may acquire original data using the electronic device’s camera (e.g., sensor module (176) of FIG. 1, camera module (180), first camera (265a, 265b), second camera (270a, 270b), third camera (245) of FIG. 2, second functional camera (311, 312), first functional camera (315), depth sensor (317) of FIG. 3a, third functional camera (328) of FIG. 3b, or fourth functional camera (325, 326)). The original data may include an image (e.g., a still image or a video) of the actual space (or physical environment) surrounding the electronic device.

[0319] According to one embodiment, the electronic device may acquire original data using the electronic device's microphone (e.g., the input module (150) of FIG. 1, the first microphone (250a), the second microphone (250b), or the third microphone (250c) of FIG. 2). The original data may include sound (or acoustic data) around the electronic device.

[0320] In operation 1120, the electronic device can generate first data at the short-term memory level by performing a first processing on the original data.

[0321] According to one embodiment, the first processing may include, for each of the images of the original data, at least one of vectorization of an object appearing in the image, generation of object information by layer of the image, or generation of metadata of the image.

[0322] According to one embodiment, an electronic device can generate a vector image by performing vectorization on an object appearing in each of the images of the original data. The vector image is information of a geometric shape and may include attributes such as the type of the shape (e.g., circle, rectangle, line, or polygon), the center coordinates of the shape, the radius of the shape, the color of the shape, the color or thickness of the outline of the shape, the coordinate array of the shape, the size of the shape, the rotation angle of the shape, and / or the transparency of the shape.

[0323] According to one embodiment, the electronic device can generate layer-specific object information for each of the images of the original data as part of a vector image. The vector image may include attributes such as object information contained in a background layer and / or object information contained in a foreground layer.

[0324] According to one embodiment, the electronic device can generate metadata for each of the images of the original data. The metadata for each image may include at least one of text describing the image, characteristic information of the image, or a prompt for generating an image similar to the image.

[0325] According to one embodiment, the electronic device can generate text describing each of the images in the original data. The electronic device can generate text describing the scene of the image, or the subject, content, or relationship of the objects included in the image, for each of the images in the original data.

[0326] According to one embodiment, the electronic device can determine characteristic information of each of the images in the original data. For example, the electronic device can determine characteristic information for each of the images in the original data, such as the width, height, and color space of the image, the location where the image was acquired, or a timestamp where the image is stored, as metadata.

[0327] According to one embodiment, the electronic device may generate a prompt for each of the images of the original data to generate an image similar to the image. For example, the prompt for each image may correspond to text describing the aforementioned image. For example, the prompt for each image may be generated based on characteristic information of the aforementioned image and / or attributes determined as a result of vectorization of the image. For example, the prompt may be text such as, 'The vector image contains a red circle with a black outline in the center and a green rectangle tilted at 45 degrees located at the upper right corner. The vector image has an sRGB color space, a width of 800 pixels, and a height of 600 pixels.'

[0328] According to one embodiment, the electronic device can store each image of the original data in the form of a raster image.

[0329] The electronic device can store at least one of a vector image, object information, metadata, or raster image generated by performing a first processing on each image of the original data as the first data.

[0330] In operation 1130, the electronic device can generate second data in a long-term memory level by performing a second processing on at least a portion of first data in a short-term memory level.

[0331] According to one embodiment, the electronic device can determine the importance of each part of the first data. Based on the importance of each part of the first data, the electronic device can perform a second processing on at least a part of the first data.

[0332] The importance of each part of the first data may reflect the time elapsed since the part was stored and / or the frequency with which it was utilized for search augmentation. The electronic device may convert a part of the first data to a long-term memory level by performing a second processing on the part of the first data that is old or not frequently utilized for search augmentation. A method for converting data at a short-term memory level to data at a long-term memory level is described in detail with reference to FIG. 12.

[0333] According to one embodiment, the second processing may include, for each of at least a portion of the images of the first data, at least one of resizing the image, merging the image with a portion of the previously generated second data, or summarizing the image.

[0334] According to one embodiment, the electronic device may perform resizing for each of at least a portion of images of the first data. For example, the electronic device may reduce the size of each image (or part) of at least a portion of the first data according to the importance of each image. For example, the electronic device may downscale the image according to the importance of each image of at least a portion of the first data.

[0335] According to one embodiment, the electronic device may perform merging between parts of previously generated second data for each of at least a portion of images of the first data. For example, the electronic device may identify parts of previously generated second data where the similarity to the corresponding image of the first data is greater than (or exceeds) a threshold, or the distance to the corresponding image of the first data is less than (or less than) a threshold, based on each image (or part) (e.g., a vector image or a raster image) of at least a portion of the first data, and the object information and / or metadata of the said image.

[0336] An electronic device may merge a portion of first data and a portion of second data where their similarity to each other is greater than or equal to a threshold, or their distance is less than or equal to a threshold. A portion of second data may be inserted into a portion of first data, or a portion of first data may be inserted into a portion of second data. According to one embodiment, the electronic device may merge both portions based on the inclusion relationship (or upper / lower category) between a portion of first data and a portion of second data. For example, if the portion of first data is an image related to 'school' and the portion of second data is an image related to 'blackboard', the electronic device may insert a portion of second data into the portion of first data.

[0337] Therefore, when the electronic device determines (or generates) session memory data to generate a response to a user's query in the future, it can construct effective session memory data by referencing a single image in which multiple images (or parts) are merged. Additionally, the electronic device can effectively manage multiple images at once.

[0338] According to one embodiment, the electronic device can perform a summary of the corresponding images for each of at least some of the images of the first data.

[0339] As described above, the electronic device can generate text describing the image for each of the images in the original data. The electronic device can generate a prompt for generating an image similar to the image for each of the images in the original data.

[0340] For example, the electronic device may summarize text for each image (or part) of at least part of the first data, or summarize a prompt for generating an image similar to the image. For each image of at least part of the first data, the electronic device may replace the existing text or prompt with the summary text generated as a result of the summary.

[0341] According to one embodiment, the electronic device may perform simplification on each of at least some of the images of the first data. For example, the electronic device may abbreviate or delete some items from each of at least some of the images (e.g., vector images or raster images), object information, or metadata of the first data. According to one embodiment, the summarization of the aforementioned images may be understood as part of the simplification operation.

[0342] FIG. 12 is a flowchart of a method for converting data at a short-term memory level to data at a long-term memory level according to one embodiment.

[0343] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0344] According to one embodiment, the following operations 1210 and 1220 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions.

[0345] According to one embodiment, the operation 620 of generating second data at a long-term memory level by performing a second processing on at least a portion of first data at a short-term memory level of FIG. 6 or the operation 1130 of generating second data at a long-term memory level by performing a second processing on at least a portion of first data at FIG. 11 may include operations 1210 and 1220.

[0346] In operation 1210, the electronic device can determine the importance of each part of the first data based on a timestamp regarding each part of the first data at a short-term memory level and the frequency with which each part of the first data is utilized for search augmentation to generate a response for the user.

[0347] As described with reference to FIG. 11, for each of the images of the original data, characteristic information such as the width, height, and color space of the image, the location where the image was acquired, or the timestamp where the image is stored can be determined and stored as metadata.

[0348] According to one embodiment, an electronic device can determine the importance of each part of the first data based on a timestamp regarding each part of the first data at a short-term memory level. For example, based on the timestamp regarding each part of the first data, the electronic device can decrease the importance by a predetermined value whenever a predetermined time has elapsed since the part was stored. For example, based on the timestamp regarding each part of the first data, the electronic device can decrease the importance linearly as time passes since the part was stored.

[0349] As described with reference to FIGS. 6 through 9, the electronic device may determine a portion of the first data at the short-term memory level as target data (or as part of the target data) based on context information of the user's query. Session memory data is determined based on the said portion, and at least a response to the user's query may be provided based on the session memory data.

[0350] According to one embodiment, the electronic device can determine the importance of each part of the first data based on the frequency with which each part of the first data at the short-term memory level is utilized for search augmentation to generate a response for a user.

[0351] For example, the electronic device may increase the importance of a portion when any portion of the first data is determined as target data (i.e., when a response is provided based on any portion of the first data). For example, the electronic device may increase the importance of a portion by the number of times that portion is determined as target data.

[0352] In operation 1220, the electronic device may generate second data by performing a second processing on at least a portion of the first data such that the corresponding importance satisfies a defined condition regarding the second processing.

[0353] According to one embodiment, a defined condition regarding the second processing may include the importance of any part of the first data being below (or lower than) a threshold. The electronic device may generate second data by performing a second processing on a part of the first data that is old or not frequently used for search enhancement. Generating second data may be understood as the part (or degraded part) generated as a result of performing the second processing on any part of the first data being stored as part of the second data at the long-term memory level.

[0354] According to one embodiment, the electronic device may perform a second processing on at least a portion of the first data when the capacity of the first data at the short-term memory level reaches a predetermined threshold. For example, the electronic device may generate second data by performing a second processing on a portion of the first data with the lowest corresponding importance at the time when the capacity of the first data at the short-term memory level reaches a predetermined threshold.

[0355] FIGS. 13a, FIGS. 13b, and FIGS. 13c are drawings illustrating a method for refining data at the short-term and long-term memory level according to one example, respectively.

[0356] As described with reference to FIG. 5, a response system (e.g., the response system (5) of FIG. 5) may include a storage (1300) (e.g., the storage (50) of FIG. 5) for retrieving related data based on a user's query. For example, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and FIG. 3b) may include a storage (1300).

[0357] According to one embodiment, the storage (1300) may include a short-term memory level (or, a sub-store of the short-term memory level) (1301) and a long-term memory level (or, a sub-store of the long-term memory level) (1303).

[0358] Referring to FIGS. 13a and 13b, the storage (1300) may contain first data of a short-term memory level (1301) (e.g., a portion (1310) or a portion (1330)) of the first data. The storage (1300) may contain second data of a long-term memory level (1303) (e.g., a portion (1320) or a portion (1340)) of the second data.

[0359] The electronic device can perform any processing (e.g., first processing, second processing and / or third processing) on ​​the original data to overcome the capacity limit of the storage (1300) and to build a data pool for generating an effective response to a user's query.

[0360] As described with reference to FIG. 11, the electronic device can generate second data of a long-term memory level (1303) by performing a second processing on at least a portion of the first data of a short-term memory level (1301).

[0361] For example, the electronic device may perform resizing on the portion (1310) according to the importance of the portion (1310) of the first data in the short-term memory level (1301). For example, the electronic device may downscale the portion (1310) according to the importance of the portion (1310) of the first data.

[0362] Referring to FIG. 13a, for example, an electronic device can perform a merger between a portion of the first data (1310) and a portion of the second data (1320) that has already been generated.

[0363] Referring to FIG. 13b, for example, an electronic device may perform a merge between a part (1330) of first data and a part (1340) of second data that has already been generated. The part (1340) of second data may be a part (1350) that has been merged within a background part (1340). Based on the importance of each part (1330) of first data and part (1350), the electronic device may perform resizing of at least some of the part (1330) and part (1350) when merging between the part (1330) of first data and part (1340) of second data. For example, the electronic device may reduce the part (1330) of greater importance more than the part (1350), and further reduce the part (1350) of less importance. The electronic device may position the part (1330) of greater importance closer to the center.

[0364] As described below with reference to FIG. 14, the electronic device can refine any part of the second data by performing a third processing on at least a part of the second data of the long-term memory level (1303).

[0365] Referring to FIG. 13c, for example, the electronic device may perform a rearrangement, resizing, or deletion of at least one of the previously merged parts (1360, 1370) within the second data part (1380). For example, if the importance of part (1380) is below (or below) a predetermined threshold, the electronic device may delete part (1380) and enlarge part (1360) to change the arrangement so that another part (1370) is inserted into part (1360). For example, if the importance of part (1370) increases, the electronic device may change the arrangement so that part (1370) is closer to the center of the second data part (1360).

[0366] FIG. 14 is a flowchart of a method for refining data at the long-term memory level according to one embodiment.

[0367] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0368] According to one embodiment, the following operations 1410 and 1420 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (301) of FIG. 3a and 3b). The electronic device may include at least some of the components of the electronic device (101) described in FIG. 1. For example, the electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) including a processing circuit. The electronic device may include a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions.

[0369] As described with reference to FIGS. 6 to 13c, an electronic device can generate first data at a short-term memory level by performing a first processing on original data acquired by the electronic device. The electronic device can generate second data at a long-term memory level by performing a second processing on at least a portion of the first data. According to one embodiment, the long-term memory level may include a plurality of long-term memory levels according to the degree of degradation. The electronic device can subdivide and refine the second data according to the plurality of long-term memory levels.

[0370] According to one embodiment, operations 1410 and 1420 may be performed after operation 1130 of FIG. 11.

[0371] In operation 1410, the electronic device may determine the importance of each part of the second data based on a timestamp regarding each part of the second data and the frequency with which each part of the second data is utilized in search augmentation to generate a response for the user.

[0372] The importance of each part of the second data may reflect the time elapsed since the part was stored and / or the frequency with which it has been utilized for search augmentation. The electronic device may refine parts of the second data that are old or not frequently utilized for search augmentation by performing a third processing on them, thereby converting them to a long-term memory level with a higher degree of degradation.

[0373] As described with reference to FIG. 11, for each of the images of the original data, characteristic information such as the width, height, and color space of the image, the location where the image was acquired, or the timestamp where the image is stored can be determined and stored as metadata.

[0374] According to one embodiment, an electronic device can determine the importance of each part of the second data based on a timestamp regarding each part of the second data at a long-term memory level. For example, based on the timestamp regarding each part of the second data, the electronic device can decrease the importance by a predetermined value whenever a predetermined time has elapsed since the part was stored. For example, based on the timestamp regarding each part of the second data, the electronic device can decrease the importance linearly as time passes since the part was stored.

[0375] As described with reference to FIGS. 6 through 9, the electronic device may determine a portion of second data at the long-term memory level as target data (or as part of target data) based on context information of a user's query. Session memory data is determined based on said portion, and at least a response to a user's query may be provided based on the session memory data.

[0376] According to one embodiment, the electronic device may determine the importance of each part of the second data based on the frequency with which each part of the second data at the long-term memory level is utilized for search augmentation to generate a response for the user. The frequency with which any part of the data is utilized for search augmentation to generate a response for the user may represent the frequency with which the part is determined as target data.

[0377] For example, the electronic device may increase the importance of a portion of the second data when any portion of the second data is determined as target data (i.e., when a response is provided based on any portion of the second data). For example, the electronic device may increase the importance of a portion by the number of times that portion is determined as target data.

[0378] In operation 1420, the electronic device may perform a third processing on at least a portion of the second data according to a long-term memory level determined based on the importance of each portion of the second data among a plurality of long-term memory levels.

[0379] According to one embodiment, the electronic device can determine a long-term memory level corresponding to the importance of each part of the second data among a plurality of long-term memory levels. For example, the electronic device can determine the part of the second data where the corresponding importance value belongs to a first value range as a first long-term memory level, and the part of the second data where the corresponding importance value belongs to a second value range as a second long-term memory level.

[0380] The number of long-term memory levels is exemplary, for example, long-term memory levels may include medium-long-term memory levels and ultra-long-term memory levels. For example, long-term memory levels may include multiple long-term memory levels according to the degree of degradation (e.g., degradation level 1 to degradation level 10), but this is not limited to the present disclosure.

[0381] The third processing may include, for each of at least some of the images of the second data, at least one of resizing the image, changing the arrangement of the merged image within the image, resizing or deleting, or deleting the image.

[0382] According to one embodiment, the electronic device may perform resizing for each of at least a portion of images of the second data. For example, the electronic device may reduce the size of each image (or part) of at least a portion of the second data according to the importance of each image. For example, the electronic device may downscale the image according to the importance of each image of at least a portion of the second data.

[0383] According to one embodiment, the electronic device may perform a rearrangement, resizing, or deletion of at least one image already merged within a portion thereof for each of the images of at least a portion of the second data. For example, if other images are inserted within any image of the second data, the electronic device may position the image closer to or further from the center, reduce or enlarge it, or delete it according to the importance of each inserted image.

[0384] According to one embodiment, as described with reference to FIG. 8, the electronic device may perform target processing on a portion of second data satisfying a predetermined condition based on a portion of first data satisfying a predetermined condition regarding a user's query. The electronic device may determine the result of the target processing as the first data at the short-term memory level. That is, the electronic device may include the restored portion of second data in the first data at the short-term memory level.

[0385] According to one embodiment, the electronic device can determine a target memory level corresponding to the importance of each part of the second data. The target memory level may be one of a short-term memory level and a plurality of long-term memory levels. If the importance of a part of the second data falls within a target value range determined for any memory level, the electronic device can determine that memory level as the target memory level of that part.

[0386] For example, the electronic device may determine the target memory level of the portion in which the corresponding importance value in the second data belongs to the first target value range as the short-term memory level. The electronic device may determine the target memory level of the portion in which the corresponding importance value in the second data belongs to the second target value range as the first long-term memory level. The electronic device may determine the target memory level of the portion in which the corresponding importance value in the second data belongs to the third target value range as the second long-term memory level.

[0387] The electronic device may include a portion of the second data in the target memory level (or data of the target memory level) if the target memory level determined for that portion of the second data is a memory level with a lower degree of degradation than the current memory level of that portion. For example, if the electronic device determines a short-term memory level as the target memory level for any portion of the second data of the long-term memory level, it may include that portion in the first data of the short-term memory level. For example, if the electronic device determines a first long-term memory level with a lower degree of degradation than the second long-term memory level as the target memory level for any portion of the second data of the second long-term memory level, it may include that portion in the second data of the first long-term memory level.

[0388] More specifically, for example, the electronic device may include a portion of the second data at the long-term memory level in the first data, while deleting it from the second data.

[0389] For example, if a portion of the second data at the long-term memory level includes multiple images, the electronic device may include some images of high importance in the first data, which are directly utilized for search augmentation. The electronic device may retain some images of relatively low importance in the second data.

[0390] According to one embodiment, the electronic device may include the restored portion of the second data in the data of the target memory level by performing a target processing to restore the portion of the second data based on the portion of the data of the target memory level (e.g., the first data or the second data).

[0391] According to one embodiment, the electronic device can include the restored portion of the second data in the data of the target memory level by performing target processing to restore the portion of the second data based on external data obtained through a web search.

[0392] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0393] According to one embodiment, an electronic device (101, 201, 301) comprises at least one processor (120) including processing circuitry; and a memory (130) including one or more storage media for storing instructions, and when instructions are executed individually or collectively by at least one processor (120), the electronic device (101, 201, 301) causes: an operation (610) to generate first data at a short-term memory level by performing a first processing on original data acquired by the electronic device (101, 201, 301); an operation (620) to generate second data at a long-term memory level by performing a second processing on at least a portion of the first data; an operation (630) to receive a user query; and an operation (640) to determine a target session for generating a response to the query based on context information of the query. The operation (650) of determining session memory data corresponding to a target session that includes at least one part of the first data or a part of the second data based on context information; the operation (660) of generating a response to a query based on at least the session memory data; and the operation (670) of outputting a response may be performed.

[0394] According to one embodiment, the operation (640) of determining a target session for generating a response to a query by at least one processor individually or based on the context information of the query may include the operation (720) of determining a session corresponding to the context information as a target session, based on the determination that among one or more sessions generated for a user, there exists a session corresponding to the context information.

[0395] According to one embodiment, the operation (640) of determining a target session for generating a response to a query based on context information of the query may include the operation (730) of generating a target session for generating a response to a query based on the determination that there is no session created for the user, or that among one or more sessions created for the user, there is no session corresponding to the context information.

[0396] According to one embodiment, the operation (650) of determining session memory data corresponding to a target session containing at least one part of a first data or a part of a second data based on context information may include: the operation (810, 910) of determining at least one memory level corresponding to a query among a short-term memory level and a long-term memory level based on context information; the operation (820) of determining target data containing at least one part of a first data or a part of a second data based on at least one memory level; and the operation (830) of determining session memory data corresponding to a target session based on the target data.

[0397] According to one embodiment, the operation (820) of determining target data including at least one of a portion of first data or a portion of second data based on at least one memory level may include the operation (920) of determining the portion of first data satisfying a predetermined condition regarding the query as target data by determining a short-term memory level as at least one memory level corresponding to the query based on context information—where the portion of first data is determined as session memory data.

[0398] According to one embodiment, the operation (820) of determining target data containing at least one of a part of first data or a part of second data based on at least one memory level may include: an operation (930) of identifying a part of second data satisfying a predetermined condition regarding a query by determining a long-term memory level as at least one memory level corresponding to a query based on context information; and an operation (960) of determining the part of first data and the part of second data as target data by determining that a part of first data satisfying a predetermined condition regarding a query exists.

[0399] According to one embodiment, the operation (820) of determining target data containing at least one of a portion of first data or a portion of second data based on at least one memory level may include: determining a long-term memory level as at least one memory level corresponding to a query based on context information, thereby identifying a portion of second data containing an object determined based on a query; and determining a portion of first data as target data by determining that a portion of first data containing an object determined based on a query exists.

[0400] According to one embodiment, the operation (830) of determining session memory data corresponding to a target session based on target data may include determining session memory data corresponding to a target session by performing a target processing to restore a part of the second data based on a part of the first data as the part of the first data is determined as target data.

[0401] According to one embodiment, the long-term memory level includes a plurality of long-term memory levels according to the degree of degradation, and the operation (820) of determining target data including at least one of a part of first data or a part of second data based on at least one memory level may include: an operation (930) of identifying a part of second data that satisfies a predetermined condition regarding the query by determining the long-term memory level as at least one memory level corresponding to the query based on context information; an operation (980) of obtaining external data regarding the query from an external source of the electronic device (101, 201, 301) by determining that the identified part of second data corresponds to a predetermined long-term memory level among the plurality of long-term memory levels where the degree of degradation is greater than a threshold; and an operation (990) of determining the external data and the part of second data as target data—session memory data corresponding to the target session is determined by performing target processing to restore the part of second data based on the external data.

[0402] According to one embodiment, the first processing may include, for each of the images of the original data, at least one of vectorization of an object appearing in the image, generation of object information by layer of the image, generation of text describing the image, or determination of characteristic information of the image.

[0403] According to one embodiment, the operation (620) of generating second data at a long-term memory level by performing a second processing on at least a portion of first data may include: an operation (1210) of determining the importance of each portion of first data based on a timestamp regarding each portion of first data and the frequency with which each portion of first data is utilized for search augmentation to generate a response for a user; and an operation (1220) of generating second data by performing a second processing on at least a portion of first data where the corresponding importance satisfies a predetermined condition regarding the second processing.

[0404] According to one embodiment, the second processing may include, for each of at least a portion of the images of the first data, at least one of resizing the image, merging the image with a portion of the previously generated second data, or summarizing the image.

[0405] According to one embodiment, the long-term memory level includes a plurality of long-term memory levels according to the degree of degradation, and when instructions are executed individually or collectively by at least one processor, the electronic device (101, 201, 301) may further perform: an operation (1410) of determining the importance of each part of the second data based on a timestamp regarding each part of the second data and the frequency with which each part of the second data is utilized for search augmentation to generate a response for a user; and an operation (1420) of performing a third processing on at least a part of the second data according to the long-term memory level determined based on the importance of each part of the second data among the plurality of long-term memory levels.

[0406] According to one embodiment, the third processing may include, for each of at least a portion of the images of the second data, at least one of resizing the image, changing the arrangement of the merged image within the image, resizing or deleting, or deleting the image.

[0407] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device (101, 201, 301) may further perform the operation of determining the result of the target processing as the first data of the short-term memory level, as determined by determining that the target processing was performed on the part of the second data based on the part of the first data.

[0408] A method performed by an electronic device (101, 201, 301) according to one embodiment may include: an operation (610) of generating first data at a short-term memory level by performing a first processing on original data obtained by the electronic device (101, 201, 301); an operation (620) of generating second data at a long-term memory level by performing a second processing on at least a part of the first data; an operation (630) of receiving a user query; an operation (640) of determining a target session for generating a response to a query based on context information of the query; an operation (650) of determining session memory data corresponding to a target session containing at least one of a part of the first data or a part of the second data based on context information; an operation (660) of generating a response to a query based on at least the session memory data; and an operation (670) of outputting a response.

[0409] According to one embodiment, an electronic device (101, 201, 301) comprises at least one processor (120) including processing circuitry; and a memory (130) including one or more storage media for storing instructions, and when instructions are executed individually or collectively by at least one processor (120), the electronic device (101, 201, 301) causes: an operation (1110) to acquire original data; and an operation (1120) to generate first data at a short-term memory level by performing a first processing on the original data. The operation (1130) of generating second data at a long-term memory level by performing a second processing on at least a portion of the first data may include: an operation (1210) of determining the importance of each portion of the first data based on a timestamp regarding each portion of the first data and the frequency with which each portion of the first data is utilized for search augmentation to generate a response for a user of an electronic device (101, 201, 301); and an operation (1220) of generating second data by performing a second processing on at least a portion of the first data where the corresponding importance satisfies a predetermined condition regarding the second processing.

[0410] According to one embodiment, the first processing may include, for each of the images of the original data, at least one of vectorization of an object appearing in the image, generation of object information by layer of the image, generation of text describing the image, or determination of characteristic information of the image.

[0411] According to one embodiment, the second processing may include, for each of at least a portion of the images of the first data, at least one of resizing the image, merging the image with a portion of the previously generated second data, or summarizing the image.

[0412] According to one embodiment, the long-term memory level includes a plurality of long-term memory levels according to the degree of degradation, and when instructions are executed individually or collectively by at least one processor, the electronic device (101, 201, 301) may further perform: an operation (1410) of determining the importance of each part of the second data based on a timestamp regarding each part of the second data and the frequency with which each part of the second data is utilized for search augmentation to generate a response for a user; and an operation (1420) of performing a third processing on at least a part of the second data according to a long-term memory level determined based on the importance of each part of the second data among the plurality of long-term memory levels—the third processing includes at least one of resizing each image, changing the arrangement of merged images within each image, or deleting for at least a part of the second data.

[0413] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device (101, 201, 301) may further perform the operation of determining the result of the target processing as the first data of the short-term memory level by determining that the target processing was performed on the second data satisfying the predetermined conditions based on the first data satisfying the predetermined conditions regarding the query.

[0414] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0415] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0416] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0417] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0418] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0419] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0420] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0421] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on the operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0422] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0423] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0424] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0425] Although the embodiments described above have been explained with reference to limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0426] Therefore, other implementations, one embodiment, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. In an electronic device (101, 201, 301), At least one processor (120) including processing circuitry; and It includes a memory (130) comprising one or more storage media for storing instructions, and When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 201, 301) is made to: An operation (610) of generating first data at a short-term memory level by performing a first processing on original data obtained by the above electronic devices (101, 201, 301); An operation (620) to generate second data at a long-term memory level by performing a second processing on at least a portion of the first data; Operation to receive user query (630); An operation (640) to determine a target session for generating a response to the above query based on context information of the above query; An operation (650) to determine session memory data corresponding to the target session that includes at least one of the part of the first data or the part of the second data based on the above context information; At least an operation (660) of generating a response to the query based on the session memory data; and The operation (670) of outputting the above response causing to perform, Electronic device (101, 201, 301).

2. In Paragraph 1, The operation (640) of determining the target session for generating a response to the query by the above instructions individually by the at least one processor or based on the context information of the query is, The operation (720) of determining that among one or more sessions created for the above user, there exists a session corresponding to the context information, and determining the session corresponding to the context information as the target session. including, Electronic device (101, 201, 301).

3. In either Paragraph 1 or Paragraph 2, The operation (640) of determining the target session for generating a response to the query based on the context information of the query is, The operation (730) of creating the target session for generating a response to the query, as determined that there is no session created for the above user, or that among one or more sessions created for the above user, there is no session corresponding to the context information. including, Electronic device (101, 201, 301).

4. In any one of paragraphs 1 through 3, The operation (650) of determining the session memory data corresponding to the target session that includes at least one of the part of the first data or the part of the second data based on the above context information is, An operation (810, 910) to determine at least one memory level corresponding to the query among the short-term memory level and the long-term memory level based on the above context information; An operation (820) for determining target data including at least one of the first data portion or the second data portion based on at least one memory level; and The operation (830) of determining the session memory data corresponding to the target session based on the above target data including, Electronic device (101, 201, 301).

5. In any one of paragraphs 1 through 4, The operation (820) of determining target data including at least one of the first data portion or the second data portion based on at least one memory level is, An operation (920) of determining a portion of the first data satisfying a predetermined condition regarding the query as the target data by determining a short-term memory level as at least one memory level corresponding to the query based on the above context information - the portion of the first data is determined as the session memory data - including, Electronic device (101, 201, 301).

6. In any one of paragraphs 1 through 5, The operation (820) of determining target data including at least one of the first data portion or the second data portion based on at least one memory level is, An operation (930) of identifying a portion of the second data that satisfies a predetermined condition regarding the query by determining a long-term memory level as the at least one memory level corresponding to the query based on the above context information; and The operation (960) of determining the first data portion and the second data portion as the target data, based on the determination that there exists a first data portion satisfying the above-determined condition regarding the above-determined query. including, Electronic device (101, 201, 301).

7. In any one of paragraphs 1 through 6, The operation (830) of determining the session memory data corresponding to the target session based on the target data is, An operation to determine the session memory data corresponding to the target session by performing target processing to restore the part of the second data based on the part of the first data, as the part of the first data and the part of the second data are determined as the target data. including, Electronic device (101, 201, 301).

8. In any one of paragraphs 1 through 7, The above long-term memory level includes a plurality of long-term memory levels according to the degree of degradation, and The operation (820) of determining target data including at least one of the first data portion or the second data portion based on at least one memory level is, An operation (930) of identifying a portion of the second data that satisfies a predetermined condition regarding the query by determining the long-term memory level as the at least one memory level corresponding to the query based on the above context information; An operation (980) of obtaining external data regarding the query from an external source of the electronic device (101, 201, 301) as it is determined that a portion of the identified second data corresponds to a predetermined long-term memory level among the plurality of long-term memory levels in which the degree of degradation is greater than a threshold; and Operation (990) of determining the portion of the above external data and the above second data as the target data - by performing target processing to restore the portion of the above second data based on the above external data, the session memory data corresponding to the target session is determined - including, Electronic device (101, 201, 301).

9. In any one of paragraphs 1 through 8, The operation (820) of determining target data including at least one of the first data portion or the second data portion based on at least one memory level is, An operation of identifying a portion of the second data containing an object determined based on the query, by determining the long-term memory level as the at least one memory level corresponding to the query based on the context information; and An operation of determining the portion of the first data as the target data, based on the determination that there exists a portion of the first data containing the object determined based on the above query. including, Electronic device (101, 201, 301).

10. In any one of paragraphs 1 through 9, The above first treatment is, For each of the images of the above original data, at least one of vectorization of an object appearing in the image, generation of object information by layer of the image, generation of text describing the image, or determination of characteristic information of the image, Electronic device (101, 201, 301).

11. In any one of paragraphs 1 through 10, The operation (620) of generating the second data of the long-term memory level by performing the second processing on at least a portion of the first data is, An operation (1210) for determining the importance of each part of the first data based on a timestamp regarding each part of the first data and the frequency with which each part of the first data is utilized for search augmentation to generate a response for the user; and The operation (1220) of generating the second data by performing the second processing on at least a portion of the first data such that the corresponding importance satisfies the predetermined conditions regarding the second processing. including, Electronic device (101, 201, 301).

12. In any one of paragraphs 1 through 11, The above second processing is, For each of at least some of the images of the first data, at least one of resizing of the image, merging of the image with some of the previously generated second data, or summarizing of the image, Electronic device (101, 201, 301).

13. In any one of paragraphs 1 through 12, The above long-term memory level includes a plurality of long-term memory levels according to the degree of degradation, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device (101, 201, 301) is caused to: An operation (1410) for determining the importance of each part of the second data based on a timestamp regarding each part of the second data and the frequency with which each part of the second data is utilized for search augmentation to generate a response for the user; and An operation (1420) of performing a third processing on at least a portion of the second data according to a long-term memory level determined based on the importance corresponding to each portion of the second data among the plurality of long-term memory levels - the third processing includes, for each of the images of at least a portion of the second data, at least one of resizing the image, changing the arrangement of merged images within the image, resizing or deleting, or deleting the image - to make it perform more, Electronic device (101, 201, 301).

14. In any one of paragraphs 1 through 13, When the above instructions are executed individually or collectively by the at least one processor, the electronic device (101, 201, 301) is caused to: When target processing is performed on a portion of the second data based on a portion of the first data, the operation of determining the result of the target processing as the first data of the short-term memory level. to make it perform more, Electronic device (101, 201, 301).

15. A method performed by an electronic device (101, 201, 301), An operation (610) of generating first data at a short-term memory level by performing a first processing on original data obtained by the above electronic devices (101, 201, 301); An operation (620) to generate second data at a long-term memory level by performing a second processing on at least a portion of the first data; Operation to receive user query (630); An operation (640) to determine a target session for generating a response to the above query based on context information of the above query; An operation (650) to determine session memory data corresponding to the target session that includes at least one of the part of the first data or the part of the second data based on the above context information; At least an operation (660) of generating a response to the query based on the session memory data; and The operation (670) of outputting the above response including, method.

Images