Electronic device, method, and recording medium for supporting multimodal service

Abstract

The present disclosure relates to an electronic device, a method, and a recording medium for supporting a multimodal service. The electronic device may comprise at least one sensor including an image sensor. The electronic device may comprise a memory including one or multiple storage media for storing instructions. The electronic device may comprise at least one processor including processing circuitry. The instructions may, when individually or collectively executed by the at least one processor, cause the electronic device to perform at least one operation. The at least one operation may include an operation for acquiring first text of a first language from a target image output by the image sensor. The at least one operation may include an operation for acquiring second text obtained by translating the acquired first text into a second language. The at least one operation may include an operation for acquiring context information about a use environment on the basis of at least one of a target image, the first text, the second text, or sensed data of at least one sensor. The at least one operation may include an operation for outputting, on the basis of the acquired context information, one or a plurality of pieces of recommended texts related to the acquired second text.

Description

Electronic device, method, and recording medium supporting multimodal services

[0001] The present disclosure relates to an electronic device, method, and recording medium that support multimodal services.

[0002] Recently, immersive technologies that allow users to experience simulated environments are being developed. Immersive technologies are changing the way users work, live, and enjoy leisure. Immersive technologies may include virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR). XR technology may be a technology that encompasses various immersive technologies, including VR, AR, or MR.

[0003] AR technology is a technology that superimposes virtual objects or information onto a real-world environment, making them appear as if they exist in the physical environment. Computing and display technologies have enabled the development of systems for AR experiences. In the aforementioned AR experience, digitally regenerated images or parts thereof can be presented to the user as if they were real, or in a manner that allows them to be perceived as real.

[0004] AR devices that provide an AR experience to users by applying AR technology can be implemented as wearable devices such as glasses, head-mounted display (HMD) devices, virtual reality headsets (VRH), and AR helmets.

[0005] Furthermore, industrialization has led to the emergence of advanced technological fields such as artificial intelligence (AI). AI technology is evolving beyond simple translation services to provide services that require complex prediction or reasoning, such as understanding human psychological states. To achieve this, it may be necessary to move beyond merely providing fragmentary functions like translation and instead offer multimodal services through integration with other services.

[0006] The information described above may be provided as related art for the purpose of aiding understanding of this document. None of the foregoing is to be claimed as prior art related to this document, nor is it to be used to determine prior art.

[0007] The present disclosure may provide an electronic device, a method, and a recording medium that support subsequent services by reusing results from multimodal processing.

[0008] As an example, the electronic device may include at least one sensor including an image sensor. The electronic device may include a memory including one or more storage media for storing instructions. The electronic device may include at least one processor including a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to perform at least one operation. The at least one operation may include an operation of obtaining a first text of a first language from a target image output by the image sensor. The at least one operation may include an operation of obtaining a second text by translating the obtained first text into a second language. The at least one operation may include an operation of obtaining context information regarding a usage environment based on at least one of the target image, the first text, the second text, or the sensing data of the at least one sensor. The at least one operation may include an operation of outputting one or more recommended texts related to the obtained second text based on the obtained context information.

[0009] As an example, a method of operation of an electronic device may include an operation of acquiring a first text of a first language from a target image output by an image sensor. The method of operation may include an operation of acquiring a second text by translating the acquired first text into a second language. The method of operation may include an operation of acquiring context information regarding a usage environment based on at least one of the target image, the first text, the second text, or the sensing data of at least one sensor. The method of operation may include an operation of outputting one or more recommended texts related to the acquired second text based on the acquired context information.

[0010] As an example, a recording medium may store instructions that can be read by a computer. When executed by at least part of at least one processor included in the electronic device, said instructions may cause said electronic device to perform at least one operation. The at least one operation may include an operation of obtaining a first text of a first language from a target image output by an image sensor. The at least one operation may include an operation of obtaining a second text that is a translation of the obtained first text into a second language. The at least one operation may include an operation of obtaining context information regarding a usage environment based on at least one of the target image, the first text, the second text, or the sensing data of the at least one sensor. The at least one operation may include an operation of outputting one or more recommended texts related to the obtained second text based on the obtained context information.

[0011] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0012] FIG. 1 is a partially exploded perspective view of an augmented reality (AR) device according to one embodiment of the present disclosure.

[0013] FIG. 2a is a front perspective view of an electronic device according to one embodiment.

[0014] FIG. 2b is a perspective view of the electronic device of FIG. 2a viewed from the rear.

[0015] FIG. 3 is a block diagram of an exemplary electronic device capable of performing the operations described in the present disclosure.

[0016] FIG. 4 is an exemplary block diagram for providing a generative artificial intelligence (AI) model in an electronic device according to one embodiment.

[0017] FIG. 5 is a block diagram of an exemplary AI system capable of performing the operations described in the present disclosure.

[0018] FIG. 6 is an exemplary block diagram of a device supporting multimodal services in an electronic device according to one embodiment.

[0019] FIG. 7 is a control flow diagram for performing a multimodal service in an electronic device according to one embodiment.

[0020] FIGS. 8a to 8h are drawings for explaining a multimodal service in conjunction with a translation result in an electronic device according to one embodiment.

[0021] FIG. 9 is a control flow diagram for performing a multimodal service in conjunction with a translation result in an electronic device according to one embodiment.

[0022] FIGS. 10a to 10d are drawings for illustrating an exemplary provision of a navigation service for a multimodal service linked to a translation result in an electronic device according to one embodiment.

[0023] FIG. 11 is a control flow diagram for performing navigation based on translation results into a multimodal service in an electronic device according to one embodiment.

[0024] FIG. 12 is an exemplary block diagram of an electronic device in a network environment according to various embodiments.

[0025] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0026] FIG. 1 is a partially exploded perspective view of an AR device (100) according to one embodiment of the present disclosure.

[0027] Referring to FIG. 1, an AR device (100) may be worn by a user. The AR device (100) may include glasses-shaped AR glasses (or smart glasses) (hereinafter referred to as 'AR glasses') worn on the user's face, or an HMD, VRH, or AR helmet worn on the user's head. Although the AR device (100) in FIG. 1 is exemplarily assumed to be AR glasses, it is understood that the examples proposed in this disclosure are not limited to AR glasses but can be commonly applied to other types or forms of AR devices (100).

[0028] The AR device (100) can render or output a display in front of the user's eyes. The AR device (100) can display visually expanded reality content through the display. The visually expanded reality content can provide visual information to the user. The AR device (100) can move the screen according to the user's eye movements (e.g., gaze) to provide a real scene or a realistic virtual image.

[0029] For example, an AR device (100) may provide an AR service that outputs at least one virtual object overlaid in an area determined to be the user's field of view (FOV). For example, the area determined to be the user's FOV may be an area determined to be perceptible to a user wearing the AR device (100). For example, the area determined to be the user's FOV may be an area that includes the whole or at least a part of the display of the AR device (100). The AR device (100) may include a plurality of transparent members corresponding to each of the user's eyes.

[0030] In one example, the AR device (100) may include a display module (117-1, 117-2), a camera (113-1, 113-2), and a support member (110-1, 110-2). The camera (113-1, 113-2) may capture an image corresponding to the user's FOV or measure the distance to an object. The camera (113-1, 113-2) may be used for head tracking or spatial recognition. The camera (113-1, 113-2) may also recognize the user's movements. The AR device (100) may include an audio output member.

[0031] The camera (113-1, 113-2) can be used to detect the movement of an object, i.e., an image corresponding to the user's FOV, or for spatial recognition. The camera (113-1, 113-2) can be used to detect the user's pupil. The camera (113-1, 113-2) can be used to track the user's pupil. The camera (113-1, 113-2) can be used to adjust the center of the virtual image projected onto the AR device (100) so that it is positioned according to the direction in which the pupil of the user wearing the AR device (100) gazes. For example, the camera (113-1, 113-2) can be a GS (global shutter) camera. A GS camera can detect the pupil and track rapid pupil movements without delay. The camera (113-1, 113-2) may include a left camera (113-1) and / or a right camera (113-2).

[0032] For example, the display module (117-1, 117-2) may include a left lens portion (140-1) and / or a right lens portion (140-2). The virtual object output through the display module (117-1, 117-2) may include information related to an application program running on the AR device (100). Additionally, the virtual object output through the display module (117-1, 117-2) may include information related to a real object located in a space / region corresponding to an area determined as the user's field of view (FOV). For example, the AR device (100) may identify an external object included in at least a portion of the image information related to the real space acquired through the camera (113-1, 113-2) that corresponds to an area determined as the user's FOV.

[0033] The AR device (100) can output a virtual object related to an external object identified in at least part of the AR device (100) through an area of ​​the display area of ​​the AR device (100) that is determined to be the user's FOV. The external object may include an object existing in real space (e.g., a real scene).

[0034] As an example, the lens portion (140-1, 140-2) may include a condensing lens, a vision correction lens, or a waveguide in a transparent member. For example, the transparent member may be formed from a glass plate, a plastic plate, or a polymer. The transparent member may be made to be completely transparent or translucent. The transparent member may include a left lens portion (140-1) facing the left eye of a user wearing the AR device (100). The transparent member may include a right lens portion (140-2) facing the right eye of a user wearing the AR device (100). If the display is transparent, a screen may be provided at a position facing the user's eyes.

[0035] The waveguide can transmit light generated from the light source of the display to the user's eye. For example, the waveguide may be located at least partially in a part of the lens portion (140-1, 140-2).

[0036] For example, the lens portion (140-1, 140-2) may include a display panel or a lens (e.g., glass). For example, the display panel may be a transparent material such as glass or plastic. The lens portion (140-1, 140-2) may be composed of a transparent element. A user may perceive the actual space behind the lens portion (140-1, 140-2) by passing through the lens portion (140-1, 140-2). The lens portion (140-1, 140-2) may display a virtual object in at least a portion of the transparent element so that it appears to the user as if the virtual object has been superimposed on at least a part of the actual space.

[0037] According to one example, the support member (110-1, 110-2) may include a printed circuit board (PCB) (114-1, 114-2) for transmitting electrical signals to each component of the AR device (100). The support member (110) may include a speaker (115-1, 115-2) for outputting audio signals. The support member (110) may include a battery (116-1, 116-2) for supplying power. For example, in an eyeglass-type AR device (100), the support member (110-1, 110-2) may be placed on a temple (111-1, 111-2). The support member (110-1, 110-2) may include a hinge member for coupling to a rim (120-1, 120-2) of the AR device (100).

[0038] The speakers (115-1, 115-2) may include a left speaker (115-1) for transmitting an audio signal to the user's left ear. The speakers (115-1, 115-2) may include a right speaker (115-2) for transmitting an audio signal to the user's right ear.

[0039] The AR device (100) may include a microphone (121) for receiving the user's voice and ambient sounds. The AR device (100) may include at least one illumination LED (112-1, 112-2) to increase the accuracy of at least one camera (113-1, 113-2) (e.g., an ET (eye tracking) camera, an extroverted camera, or a recognition camera). For example, the illumination LED (112-1, 112-2) may be used as an auxiliary device to increase accuracy when photographing the user's pupil with the camera (113-1, 113-2). The illumination LED (112-1, 112-2) may use an IR LED of an infrared wavelength rather than a visible light wavelength. For example, the light emitter (112-1, 112-2) can be used as an auxiliary device when capturing or photographing a user's gesture with a camera (113-1, 113-2), when it is not easy to detect the subject due to dark lighting.

[0040] In the present disclosure, an "augmented reality (AR) system" refers to a system that displays a virtual image together with a real-world physical environment space or displays a real-world object and a virtual image together.

[0041] In the present disclosure, an "augmented reality device (AR device)" is a device capable of expressing "augmented reality" and may include an augmented reality glasses device in the shape of glasses worn by a user on the face, a head-mounted display (HMD) device worn on the head, a virtual reality headset (VRH), or an augmented reality helmet.

[0042] In the present disclosure, a "real scene" is a scene of the real world viewed by a user through an augmented reality device and may include real-world objects. A "virtual image" is an image generated through an optical engine and may include both static and dynamic images. The virtual image may be observed together with the real scene and may be an image displaying information about real-world objects within the real scene, information about the operation of the augmented reality device, or control menus.

[0043] According to one example, an augmented reality device comprises an optical engine for generating a virtual image composed of light generated from a light source, and a waveguide formed of a transparent material to guide the virtual image generated by the optical engine to the user's eyes and to allow the user to view a scene of the real world at the same time. Since the augmented reality device must be able to observe a scene of the real world at the same time, an optical element is required to change the path of light that basically has straight-line propagation in order to guide the light generated by the optical engine to the user's eyes through the waveguide. At this time, the light path may be changed by using reflection by a mirror, etc., or by diffraction by a diffractive element such as a DOE (diffractive optical element) or HOE (holographic optical element), but is not limited thereto.

[0044] FIG. 2a is a front view of an electronic device according to one embodiment, and FIG. 2b is a rear view of the electronic device of FIG. 2a.

[0045] Referring to FIG. 2a or FIG. 2b, the electronic device (200) may include a housing (210) comprising a first surface (or front) (210A), a second surface (or rear) (210B), or a side (210C) surrounding the space between the first surface (210A) and the second surface (210B). For example, the housing may refer to a structure forming some of the first surface (210A), the second surface (210B), and the side (210C) of FIG. 2. The first surface (210A) may be formed by a front plate (202) (e.g., a glass plate or a polymer plate including various coating layers) in which at least a portion is substantially transparent. The second surface (210B) may be formed by a rear plate (211) that is substantially opaque. The rear plate (211) may be formed by, for example, coated or colored glass, ceramic, polymer, metal (e.g., aluminum, stainless steel (STS), or magnesium), or a combination of at least two of the above materials. The side (210C) may be formed by a side bezel structure (or “side member”) (218) comprising metal and / or polymer, which is combined with the front plate (202) and the rear plate (211). The rear plate (211) or the side bezel structure (218) may be formed integrally and may comprise the same material (e.g., a metallic material such as aluminum).

[0046] The front plate (202) may include two first regions (210D) that are curved seamlessly from the first surface (210A) toward the rear plate (211) at both ends of the long edge of the front plate (202). In the illustrated example (see FIG. 2), the rear plate (211) may include two second regions (210E) that are curved seamlessly from the second surface (210B) toward the front plate (202) at both ends of the long edge. For example, the front plate (202) (or the rear plate (211)) may include only one of the first regions (210D) (or the second regions (210E)). For example, the front plate (202) (or the rear plate (211)) may not include some of the first regions (210D) or the second regions (210E). For example, when viewed from the side of the electronic device (200), the side bezel structure (218) may have a first thickness (or width) on the side that does not include the first regions (210D) or the second regions (210E), and a second thickness that is thinner than the first thickness on the side that includes the first regions (210D) or the second regions (210E).

[0047] For example, the electronic device (200) may include at least one of a display (201), an audio module (203, 207, 214), a sensor module (204, 216, 219), a camera module (205, 212, 213), a key input device (217), a light-emitting element (206), a pen input device (220), and a connector hole (208, 209). The electronic device (200) may omit at least one of the components (e.g., a key input device (217), or a light-emitting element (206)) or additionally include other components.

[0048] The display (201) may be exposed, for example, through a significant portion of the front plate (202). For example, at least a portion of the display (201) may be exposed through the front plate (202) forming the first surface (210A) and the first area (210D) of the side (210C). In some embodiments, the corners of the display (201) may be formed to be largely identical to the adjacent outer shape of the front plate (202). In other embodiments (not shown), to expand the area where the display (201) is exposed, the gap between the outer edge of the display (201) and the outer edge of the front plate (202) may be formed to be largely identical.

[0049] In another embodiment (not shown), a recess or opening may be formed in a part of the screen display area of ​​the display (201), and at least one of an audio module (214), a sensor module (204), a camera module (205), and a light-emitting element (206) may be included that are aligned with said recess or said opening. In another embodiment (not shown), at least one of an audio module (214), a sensor module (204), a camera module (205), a fingerprint sensor (216), and a light-emitting element (206) may be included on the back surface of the screen display area of ​​the display (201). In another embodiment (not shown), the display (201) may be combined with or adjacent to a touch detection circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and / or a digitizer that detects a magnetic field type stylus pen. In some embodiments, at least a portion of the sensor module (204, 219) and / or at least a portion of the key input device (217) may be placed in the first regions (210D) and / or the second regions (210E).

[0050] The audio module (203, 207, 214) may include a microphone hole (203) and a speaker hole (207, 214). A microphone for acquiring external sound may be placed inside the microphone hole (203), and in some embodiments, a plurality of microphones may be placed to detect the direction of sound. The speaker hole (207, 214) may include an external speaker hole (207) and a receiver hole (214) for calls. In some embodiments, the speaker hole (207, 214) and the microphone hole (203) may be implemented as a single hole, or a speaker may be included without the speaker hole (207, 214) (e.g., a piezo speaker).

[0051] The sensor modules (204, 216, 219) can generate electrical signals or data values ​​corresponding to an internal operating state of the electronic device (200) or an external environmental state. The sensor modules (204, 216, 219) may include, for example, a first sensor module (204) (e.g., proximity sensor) and / or a second sensor module (not shown) (e.g., fingerprint sensor) disposed on a first surface (210A) of the housing (210), and / or a third sensor module (219) (e.g., HRM sensor) and / or a fourth sensor module (216) (e.g., fingerprint sensor) disposed on a second surface (210B) of the housing (210). The fingerprint sensor may be placed on the first surface (210A) (e.g., display (201)) of the housing (210) as well as on the second surface (210B). The electronic device (200) may further include at least one of a sensor module not illustrated, e.g., a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor (204).

[0052] The camera module (205, 212, 213) may include a first camera device (205) disposed on a first surface (210A) of the electronic device (200), a second camera device (212) disposed on a second surface (210B), and / or a flash (213). The camera devices (205, 212) may include one or more lenses, an image sensor, and / or an image signal processor. The flash (213) may include, for example, a light-emitting diode or a xenon lamp. In some embodiments, two or more lenses (infrared camera, wide-angle and telephoto lenses) and image sensors may be disposed on one surface of the electronic device (200).

[0053] A key input device (217) may be placed on a side (210C) of the housing (210). In another embodiment, the electronic device (200) may not include some or all of the aforementioned key input devices (217), and the key input devices (217) that are not included may be implemented in other forms, such as soft keys, on the display (201). In some embodiments, the key input device may include a sensor module (216) placed on a second side (210B) of the housing (210).

[0054] The light-emitting element (206) may be disposed, for example, on a first surface (210A) of the housing (210). The light-emitting element (206) may, for example, provide state information of the electronic device (200) in the form of light. In another embodiment, the light-emitting element (206) may, for example, provide a light source that is coupled with the operation of the camera module (205). The light-emitting element (206) may include, for example, an LED, an IR LED, and a xenon lamp.

[0055] The connector holes (208, 209) may include a first connector hole (208) capable of receiving a connector (e.g., a USB connector) for transmitting and receiving power and / or data with an external electronic device, and a second connector hole (e.g., an earphone jack) (209) capable of receiving a connector for transmitting and receiving audio signals with an external electronic device.

[0056] A pen input device (220) (e.g., a stylus pen) can be guided into the interior of the housing (210) through a hole (221) formed on the side of the housing (210) and inserted or removed, and may include a button to facilitate removal. The pen input device (220) may have a separate resonant circuit built in so as to be coupled with an electromagnetic induction panel (e.g., a digitizer) included in the electronic device (200). The pen input device (220) may include an EMR (electro-magnetic resonance) method, an AES (active electrical stylus) method, and an ECR (electric coupled resonance) method.

[0057] FIG. 3 is a block diagram of an exemplary electronic device (300) capable of performing the operations described in the present disclosure.

[0058] Referring to FIG. 3, the electronic device (300) may be one of various forms of electronic devices, such as a notebook (390), smartphones (391) having various form factors (e.g., a bar-type smartphone (391-1) (e.g., the mobile electronic device (200) of FIG. 2a and 2b), a foldable-type smartphone (391-2), or a sliderable (or rollable)-type smartphone (391-3)), a tablet (392), an AR device (e.g., the AR device (100) of FIG. 1), a cellular phone (not shown), and other similar computing devices (not shown). The components, their relationships, and their functions illustrated in FIG. 3 are illustrative only and are not intended to limit the implementations described or claimed herein. The electronic device (300) may be referred to as a mobile device, a user device, a multifunction device, a portable device, or a server.

[0059] The electronic device (300) may include components comprising at least one processor (310) (hereinafter referred to as 'processor (310)'), at least one memory (320) (hereinafter referred to as 'memory (320)'), at least one display (340) (hereinafter referred to as 'display (340)'), at least one image sensor (350) (hereinafter referred to as 'image sensor (350)'), at least one communication circuit (360) (hereinafter referred to as 'communication circuit (360)'), and / or at least one sensor (370) (hereinafter referred to as 'sensor (370)'). The components are merely exemplary. For example, the electronic device (300) may include other components (e.g., power management integrated circuitry (PMIC), audio processing circuit, antenna, rechargeable battery, or input / output interface). For example, some components may be omitted from the electronic device (300). For example, some components may be integrated into a single component. For example, the electronic device (300) may further include at least some of the unillustrated components and / or functions. At least some of each component of the illustrated (or unillustrated) electronic device may be operatively, functionally, and / or electrically connected.

[0060] The processor (310) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing operations. The processor (310) may include at least one electrical circuit and may process instructions (or programs, data) stored in memory (320) individually or collectively in a distributed manner. The processor (310) may include a processor assembly comprising one or more processing circuits. The processor (310) may include any processing circuit that is operative to control the performance and operations of one or more components of the electronic device (300) (e.g., memory (320), display (340), image sensor (350), communication circuit (360), and / or sensor (370)). For example, the processor (310) (e.g., application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or chipset). For example, the processor (310) may be implemented with a plurality of cores (or at least one core circuit), a plurality of chips, or a plurality of chipsets. For example, the processor (310) may include one or more processing circuits. For example, the processor (310) may include one or more processing circuits configured to perform the various functions of the present disclosure individually and / or collectively. As an example without limitation, at least a portion of the processor (310) may be included in a first chip of the electronic device (300), and at least another portion of the processor (310) may be included in a second chip of an electronic device different from the first chip of the electronic device (300).

[0061] For example, the processor (310) may include a central processing unit (CPU) (311), a graphics processing unit (GPU) (312), a neural processing unit (NPU) (313), an image signal processor (ISP) (314), a display controller (315), a memory controller (316), a storage controller (317), a communication processor (CP) (318), and / or a sensor interface (319). These components of the processor (310) are merely exemplary. For example, the processor (310) may include other components. For example, some components of the processor (310) may be omitted from the processor (310). For example, some components of the processor (310) may be included as separate components of the electronic device (300) outside of the processor (310). For example, some components of the processor (310) (e.g., memory controller (316)) may be included in other components (e.g., at least part of memory (320), an interface (e.g. available for connection to at least one component of the electronic device (300)), a display (340) and / or an image sensor (350)).

[0062] The number of processors (310) may be one or more. For example, the processor (310) may have the structure of a multi-core processor such as a dual core, quad core, or hexa core.

[0063] The processor (310) can control the operations of the electronic device (300) by executing instructions stored in the memory (320). For example, the processor (310) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0064] The processor (310) may cause other components of the electronic device (300) to perform various operations by executing instructions stored in memory (320). The CPU (311) (or central processing circuit) may be configured to control the components of the processor (310) based on the execution of instructions stored in memory (320) (e.g., volatile memory (312) and / or non-volatile memory (312)). The GPU (312) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The NPU (313) (or neural processing circuit, or artificial intelligence (AI) chip) may be configured to execute operations for an AI model (e.g., convolution computation). An ISP (314) (or image signal processing circuit) may be configured to process a raw image acquired through an image sensor (350) into a format suitable for a component within the electronic device (300) or a component of the processor (310). A display controller (315) (or display control circuit, or DPU (display processing unit)) may be configured to process an image acquired from a CPU (311), GPU (312), ISP (314), or memory (320) (e.g., volatile memory (312)) into a format suitable for a display (340). A memory controller (316) (or memory control circuit) may be configured to control reading data from the volatile memory (312) and writing data to the volatile memory (312). A storage controller (317) (or storage control circuit) may be configured to control reading data from the non-volatile memory (312) and writing data to the non-volatile memory (312).The CP (318) (communication processing circuit) may be configured to process data obtained from a component of the processor (310) into a format suitable for transmitting to another electronic device via the communication circuit (360), or to process data obtained from another electronic device via the communication circuit (360) into a format suitable for processing by the component of the processor (310). For example, the communication circuit (360) may include one or more communication circuits. The sensor interface (319) (or sensing data processing circuit, sensor hub) may be configured to process data regarding the state of the electronic device (300) and / or the state around the electronic device (300), obtained through the sensor (370), into a format suitable for the component of the processor (310).

[0065] Memory (320) may include one or more storage media (or one or more storage devices). For example, memory (320) may include a memory assembly comprising one or more storage media. For example, one or more storage media may include a hard drive, a permanent memory such as flash memory, ROM (read-only memory) (e.g., non-volatile memory (312)), a semi-permanent memory such as RAM (random access memory) (e.g., volatile memory (312)), any other suitable type of storage (or storage assembly), or any combination thereof. Memory (320) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (300). As an example not limited to, the cache memory may be included within the processor (310). The memory (320) may be fixedly embedded within the electronic device (300) or incorporated into one or more suitable types of components (e.g., a SIM (subscriber identity module) card and / or an SD (secure digital) card) that can be repeatedly inserted into and removed from the electronic device (300).

[0066] For example, memory (320) may store one or more software applications, such as operating system (or system) software applications, firmware software applications, driver software applications, plugin (e.g., add-in, add-on, and / or applet) software applications, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (310). For example, memory (320) may store instructions that can be called by an application programming interface (API). For example, memory (320) may store instructions within a library.

[0067] According to one example, the electronic device (300) may run at least one instance of an AI model. The instance may be an object corresponding to a program (or application), such as an AI model, for example. The instance may be named a replica, a pod, a container, or a virtual machine, and there are no restrictions on the name. The number of instances may correspond to the size of a resource (e.g., a GPU (312) or an NPU (313)), and accordingly, the number of instances may be used interchangeably with the size of the resource, or instances may be used interchangeably with the resource.

[0068] For example, multiple user requests may be input into an electronic device (300). The user requests may be associated with a service. The user requests may be processed by a first instance of a first AI model, and a first processing result may be provided from the first instance of the first AI model. The first processing result may be processed by a first instance of a second AI model, and accordingly, a second processing result may be provided by the first instance of the second AI model. Through the sequential processing of processing results, the first instance of the M AI model may receive and process the N-1 processing result. The first instance of the M AI model may provide the N processing result as a response. Accordingly, a response corresponding to the user request may be provided.

[0069] Based on the process described above, responses corresponding to each of multiple user requests may be provided. Meanwhile, since processing must be performed by an instance, the time taken to provide responses corresponding to each of multiple user requests (hereinafter referred to as “response time”) may be relatively long. Response time may affect the latency in the instance. To reduce response time, the electronic device (300) may increase the number of instances of at least one AI model, which may be referred to as scaling out. However, there may be a limit to increasing the number of instances due to hardware and / or software constraints of the electronic device (300) and / or limitations of parameters of the AI ​​model (e.g., large language model (LLM)).

[0070] FIG. 4 is an exemplary block diagram for providing a generative artificial intelligence (AI) model (400) (hereinafter referred to as 'AI system (400)') in an electronic device according to one embodiment (e.g., the electronic device (300) of FIG. 3) (hereinafter referred to as 'electronic device (300)').

[0071] Referring to FIG. 4, the AI ​​system (400) in the electronic device (300) may include a processor (310) (e.g., the processor (310) of FIG. 3), a memory (320) (e.g., the memory (320) of FIG. 3), and / or an interface (I / F) (420). According to one example, all or part of the operations performed in the AI ​​system (400) may be performed in one or more external electronic devices. For example, when the AI ​​system (400) needs to perform a function or service automatically or in response to a request from a user or another device, the AI ​​system (400) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself, or additionally. One or more external electronic devices that receive the request may perform at least part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the AI ​​system (400). The AI ​​system (400) can process the above results as they are or additionally and provide them as at least part of the response to the request.

[0072] According to one example, the AI ​​system (400) can perform services by linking one or more AI models. The AI ​​system (400) may be implemented, for example, as a single entity or as multiple entities. For example, the AI ​​system (400) may operate in an on-device environment, and there are no restrictions on its implementation form. The AI ​​system (400) operating in an on-device environment may include one or more AI models using machine learning and / or neural networks. For example, the AI ​​system (400) may execute an instance of at least one AI model. The instance may be an object corresponding to a program (or application), such as an AI model, for example, and may be named a replica, pod, container, or virtual machine, and there are no restrictions on its name. The number of instances may correspond, for example, to the size of a resource (e.g., GPU), and accordingly, the number of instances may be used interchangeably with the size of the resource, or instances may be used interchangeably with the resource. In the description to be made below, for example, it is assumed that one instance is executed for each of at least one AI model. As an instance corresponding to a specific AI model is executed, a resource of a predetermined size may be utilized. A resource of a predetermined size may mean, for example, a part of a processor (310), memory (320), and / or an I / F (420).

[0073] The AI ​​system (400) may be based on natural language processing (NLP). NLP is a technology that allows the AI ​​system (400), for example, to understand or process input of natural language that can be expressed as voice and / or text. Through NLP, the AI ​​system (400) can understand natural language and, based on this, grasp human intent or convey information in a language that humans can understand. To understand human language, NLP can learn the order of words or tokens to predict the probability of the next word or token in a given text. A token is a basic unit for processing or understanding prompts in an AI model. Key technologies of NLP include tokenization of prompts corresponding to user input, part-of-speech tagging, syntactic analysis, named entity recognition, or sentiment analysis.

[0074] The I / F (420) receives a query (430) and can transmit the received query (430) to the processor (310). The query (430) may correspond, for example, to one or multiple user requests. The query (430) may serve as a medium that guides the AI ​​system (400) to perform a task or generate a result in a desired direction. The query (430) may be the only channel through which a user can communicate with the AI ​​system (400). The query (430) needs to be clear and specific in order to obtain an answer from the AI ​​system (400) that is close to the desired result. According to one example, the I / F (420) receives a response (440) (e.g., summary message and / or response message) processed by the AI ​​system (400) based on a query (430) (e.g., received messages) and can output a response (440) converted into natural language in a form that can be perceived by humans (e.g., text, image, audio, or video). The I / F (420) can receive input or output natural language in the form of voice and / or text with at least one component, such as, for example, a keyboard, a touch panel, a display, and / or a speaker.

[0075] The processor (310) can control at least one other electrically connected component (e.g., hardware or software component) by executing software (e.g., a program). The processor (310) can perform various data processing or operations. As at least part of the data processing or operations, the processor (310) can store instructions or data received from another component (e.g., an I / F (420)) in memory (320) (e.g., volatile memory, but without limitation). As at least part of the data processing or operations, the processor (310) can process instructions or data stored in memory (320) (e.g., volatile memory, but without limitation). As at least part of the data processing or operations, the processor (310) can store the resulting data from processing instructions or data in memory (320) (e.g., non-volatile memory, but without limitation).

[0076] The memory (320) may store various data used by at least one component (e.g., processor (310) and / or I / F (420)). The data may include, for example, software (e.g., programs) and input or output data for related instructions. The memory (320) may also store at least one AI model (e.g., sLM (small LM), LLM, LVM (large vision models), LMM (large multimodal models)) for executing one or more instances.

[0077] Memory (320) can store at least one instruction. Processor (310) can execute at least one instruction stored in memory (320). When executed by processor (310), at least one instruction may cause the AI ​​system (400) to perform at least one operation. For example, as at least one instruction is executed by processor (310), at least one other component may be controlled, and / or various data processing or operations may be performed. An operation performed by processor (310) may mean that the operation is performed by (or controlled by) one entity included in processor (310), for example, which may be a main processor, but is not limited to. An operation performed may mean that a specific operation is performed by (or controlled by) multiple entities, for example, multiple processors. The execution of multiple operations may mean that all of the multiple operations are performed by (or by control of) one entity, for example, which may be a main processor (e.g., CPU (311) in FIG. 3, but is not limited to). The execution of multiple operations may mean that some of the multiple operations are performed by at least one entity, and some of the remaining operations are performed by at least one other entity. At least one instruction that causes the execution of one or more operations may be stored, for example, in one memory, or distributed and stored in each of multiple memories.

[0078] The AI ​​system (400) may share resources (e.g., data processing or computational power) corresponding to part or all of at least one processor included in the processor (310) and / or resources (e.g., data recording area) corresponding to part or all of the memory (320). For example, the AI ​​system (400) may be operated by at least one of a CPU (311), a GPU (312), or an NPU (313). The AI ​​system (400) may be operated by the CPU (311) alone, for example, by allocating at least a portion of the memory (320). The AI ​​system (400) may be operated by the GPU (312) alone, for example, by allocating at least a portion of the memory (320). The AI ​​system (400) may be operated by the NPU (313) alone, for example, by allocating at least a portion of the memory (320). The AI ​​system (400) can be performed by the CPU (311) and GPU (312) cooperating, for example, by allocating at least a portion of the memory (320). The AI ​​system (400) can be performed by the CPU (311) and NPU (313) cooperating, for example, by allocating at least a portion of the memory (320). The AI ​​system (400) can be performed by the GPU (312) and NPU (313) cooperating, for example, by allocating at least a portion of the memory (320). The AI ​​system (400) can be performed by the CPU (311), GPU (312), and NPU (313) cooperating, for example, by allocating at least a portion of the memory (320). Various embodiments described below in this disclosure are not limited to combinations of components for performing the AI ​​system (400) and may be implemented and / or applied based on any combination.

[0079] FIG. 5 is a block diagram of an exemplary AI system (500) (e.g., the AI ​​system (400) of FIG. 4) capable of performing the operations described in the present disclosure. The AI ​​system (500) may be a generative AI system, but will be referred to as 'AI system (500)' below.

[0080] The AI ​​system (500) in FIG. 5 may include an on-device AI system (e.g., the generative AI model (400) in FIG. 4) or a cloud AI system. For example, the on-device AI system may be an AI system capable of processing information on its own within an electronic device (e.g., the electronic device (300) in FIG. 3) without needing to be connected to a server or the cloud. For example, the cloud AI system may be an AI system formed by a combination of AI technology and a cloud computing architecture, capable of providing AI services and / or computing capabilities via the cloud.

[0081] As an example, the present disclosure assumes an on-device AI system; however, the proposed examples are not limited to on-device AI systems and can also be implemented through a cloud AI system or through the combination of an on-device AI system and a cloud AI system. For example, the on-device AI system may be implemented in one of various forms of electronic devices, such as a laptop (e.g., laptop (390) of FIG. 3), smartphones having various form factors (e.g., smartphones (391) of FIG. 3), a tablet (392), an AR device (e.g., AR device (100) of FIG. 1), a cellular phone (not shown), and other similar computing devices (not shown). Smartphones having various form factors may include, for example, a bar-type smartphone (391-1) (e.g., mobile electronic device (200) of FIG. 2a and 2b), a foldable-type smartphone (391-2), or a sliderable (or rollable)-type smartphone (391-3). The components, their relationships, and their functions illustrated in FIG. 5 are merely illustrative and are not intended to limit the implementations described or claimed herein.

[0082] Referring to FIG. 5, the AI ​​system (500) may include a User Query / Response Interface (510) (e.g., I / F (420) of FIG. 4) (hereinafter referred to as 'I / F (510)'), an AI framework (520), a generative AI model (530), a database (540), or an Application / Service Component (550).

[0083] The I / F (510) may receive data acquired or generated by an electronic device (e.g., the electronic device (300) of FIG. 3) or user input. Data acquired or generated by the electronic device (300) may include image or video data generated using a processor (e.g., the processor (310) of FIG. 3 or FIG. 4), values ​​received through a sensor or sensor hub (e.g., external illumination, angle of the terminal, temperature of the display (e.g., the display (340) of FIG. 3) or the electronic device (300), size or expansion / reduction information of the display (340), and captured images from an image sensor (e.g., the image sensor (350) of FIG. 3). User input may be in the form of natural language, touch coordinates or stylus coordinates acquired through a touch panel or digitizer included in the display (340), images and / or videos, but is not limited thereto. Additionally, context information may be transmitted along with the user input. Contextual information may include various additional information at the time of user input. For example, additional information may include information about the application the user is currently using or the user's location information. Furthermore, user input may take the form of a mixture of the aforementioned natural language, images, sounds, and contextual information. Additionally, user input may take the form of non-natural language, such as selecting a menu.

[0084] I / F (510) can output results of analyzing the output and / or input of the AI ​​system (500). The output may be in the form of natural language or specific content. The output may also be provided in the form of an action requested by the user. The output may also be provided in the form of a specific value specified by the user. I / F (510) can output results of the generative AI system (500) to the user. The output may be in the form of natural language or specific content. The output may also be provided in the form of an action requested by the user.

[0085] The AI ​​framework (520) can receive user input and coordinate and control each component necessary to perform the user's intent based on the user's query. For example, the AI ​​framework (520) may include a prompt design component (521), an API / Plug-in management component (523), or an output modification component or refiner component (525).

[0086] User input received from the I / F (510) can be transmitted to a prompt design component (521). The prompt design component (521) can be used to generate prompts suitable for inputting user input into a generative AI model (530) (e.g., LLM, LVM, or LMM). The prompt design component (521) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. Based on user input, the prompt design component (521) can generate prompts by accessing user preference data (543), a prompt library (541), and a knowledge component containing prompt examples, and can transmit the generated prompts to the generative AI model (530), which is an LLM or LMM.

[0087] The API / plugin management component (523) can perform the role of communicating with external information when there is a request for additional information when user input is passed as input to the generative AI model (530). The API / plugin management component (523) establishes a channel to communicate with the outside of the AI ​​interface through the API, and enables access to various data sources (e.g., knowledge repositories (545)) through the established channel.

[0088] The API / plugin management component (523) can request the application / service component (523) via the API to perform the action that ultimately performs user input, rather than an intermediate result, when the application or service needs to perform such action. The information obtained from the outside can be used to generate a prompt in the prompt design component (521) along with the user input, or can be passed as input to the generative AI model (530).

[0089] The output adjustment component (525) (or refiner component) can finely tune or reprocess the output from the generative AI model (530). The output adjustment component (525) can, for example, verify whether the content generated by the generative AI model (530) is irrelevant, contains biased content, or contains harmful content. The output adjustment component (525) can determine the extent to which the output matches the result desired by the user and, if additional processing is required, proceed with that process. Additionally, the output adjustment component (525) can configure and provide hints to the user to avoid unwanted output.

[0090] A generative AI model (530) generally refers to an AI neural network that generates new forms of data based on user input information. A generative AI model (530) may include a model that generates images and / or a model that generates language. A model that generates images may include, for example, a generative adversarial network (GAN) or a variational auto encoder (VAE). A model that generates images may be a diffusion-based AI model that uses, for example, a VAE and a transformer structure. A model that generates language may be a model trained to output the statistically most appropriate output value based on input values. Representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. Additionally, LMM is an AI model (530) capable of recognizing various forms of data input, such as text, images, voice, and video, and generating new data corresponding to them.

[0091] According to one example, the electronic device (300) may include a natural language recognition module, an NLP module, or a planner module for performing a conversation with a user using natural language. The NLP module may be an AI model pre-trained to enable the machine, the electronic device (300), to perform a conversation with a user using natural language. The NLP module may be a software module implemented, for example, by executing instructions by at least one processor (e.g., the processor (310) of FIG. 3 or FIG. 4). Hereinafter, the actions performed by the NLP module may be referred to as actions that may cause the electronic device (300) to perform when the instructions are executed individually or collectively by at least one processor (310). The NLP module may, for example, obtain a character recognition result (e.g., data converted from one or more phonemes included in a sentence entered by a user). The NLP module may provide the processing result of the character recognition result to the planner module. The processing result based on character recognition may include an intent, a target device (e.g., information about the target device), a capsule, or a combination thereof.

[0092] According to one example, an NLP module can determine the user's intent by interpreting natural language recognition results (e.g., syntactic analysis and / or semantic analysis). A natural language understanding model can identify the meaning of words extracted from natural language recognition results using linguistic features (e.g., grammatical elements) of morphemes or phrases, and determine the user's intent based on the identified meaning of the words and / or other parameters (e.g., domains or categories associated with the words). Syntactic analysis may include the act of dividing user input (e.g., user text input) into grammatical units (e.g., words, phrases, and / or morphemes) and identifying the grammatical elements possessed by the divided units. Semantic analysis may be performed through semantic matching, rule matching, and / or formula matching. Here, data transformed from one or more phonemes may represent one or more words included in the sentence entered by the user, and / or tokens for each of one or more words. Intent may be data used by the natural language platform to generate a plan. Intent may include goals and / or parameters. Goals may be used in the planner module to specify the final objective of the plan. Parameters may be values ​​input to one or more actions included in the plan in the planner module.

[0093] FIG. 6 is an exemplary block diagram of a device (600) that supports multimodal services in an electronic device (e.g., the electronic device (300) of FIG. 3) according to one embodiment (hereinafter referred to as 'multimodal device (600)').

[0094] The block configurations illustrated in FIG. 6 may include units implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of a component or part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC). Meanwhile, in the description with reference to FIG. 6, the multimodal device (600) refers to a device composed of components that perform the corresponding operation in the electronic device (300), and may substantially also be referred to as the electronic device (300).

[0095] Referring to FIG. 6, the multimodal device (600) may be an AR translation system capable of generating additional interfaces based on user task analysis information. The multimodal device (600) may support a translation function that translates text to text and / or a multimodal technology that translates text extracted from an image into text and then reuses it to perform linked services through specific tasks. For example, the multimodal device (600) may acquire a first text written in a first language from a target image of a specific object (e.g., a preview image or a captured image) by a camera, translate the acquired first text into a second text written in a second language, and use the translated second text to provide additional functions such as speech, search, or directions for conversation with a counterpart. The multimodal system (600) may, for example, translate instructions or food safety tags on a medicine bottle into a language desired by the user and provide a schedule management service to guide the timing of medication based on information contained in the translated text. For example, the multimodal system (600) can translate instructions or food safety tags on a medicine bottle into a language desired by the user and provide a schedule management service to guide the timing of medication administration based on the information contained in the translated text. For example, the multimodal system (600) can translate text contained in an image of a menu of a specific restaurant, suggest a recommended sentence considering the possibility of speech by the user based on the information contained in the translated text, or output the recommended sentence as an audible signal. The multimodal system (600) can operate to allow the user to use an application that supports only a specific language (e.g., English, Japanese, Arabic) without inconvenience. For example, while traveling, the user can receive directions to their destination by utilizing an application that supports only a specific language (e.g., English, Japanese, Arabic).

[0096] As an example, the multimodal device (600) may include an automatic speech recognition (ASR) module (610), a camera driving module (620), a sensor module (630), an image analysis module (640), a context analysis module (650), an LLM (660), an interface output module (670), or a text-to-speech (TTS) module (680).

[0097] The ASR module (610) may provide natural language processing (NLP) based on AI and / or machine learning algorithms for interaction between the user and the multimodal device (600). For example, the ASR module (610) may convert commands and / or questions entered by the user in natural language, such as voice, into machine language that the multimodal device (600) can recognize. For example, when the ASR module (610) receives voice input from a user, such as “Translate the text in the image into Korean,” it can convert the voice command entered by the user into machine language in response to enable the corresponding AI model (e.g., LLM (660)) to perform a translation service, and provide it to the corresponding AI model, LLM (660), image analysis module (640), and / or context analysis module (650). For example, the ASR module (610) can convert surrounding audible signals into machine language to obtain context information regarding the user’s current state and provide it to the LLM (660), image analysis module (640), and / or context analysis module (650).

[0098] The camera driving module (620) can drive the camera (or image sensor (e.g., the image sensor (350) of FIG. 3)) to acquire an image of a specific subject (e.g., a preview image or a captured image). For example, the camera driving module (620) can drive the camera to acquire a target image when a translation need by a user is predicted based on information acquired by the ASR module (610).

[0099] The sensor module (630) may include at least one sensor. For example, the sensor module (630) may include at least one sensor that provides an image (e.g., a preview image or a captured image) or provides sensing data that can be considered for obtaining context information regarding the user's intention. The sensor module (630) may include, for example, an image sensor whose operation can be controlled by the camera driving module (620). The sensor module (630) may include, for example, a location sensor such as a GPS that can provide sensing data regarding location information.

[0100] The image analysis module (640) can analyze a target image obtained from a camera by the camera driving module (620) and output the analysis results. For example, the image analysis module (640) can analyze information regarding an object included in the target image and output the information obtained through the analysis. The image analysis module (640) can determine user environment information based on text information included in the target image or sensing data obtained by the sensor module (630), or define tasks required by the user in that environment. For example, if the target image is an image of a menu of a specific restaurant, the image analysis module (640) can analyze the menu image included in the menu and provide information regarding the types and / or prices of food available for order.

[0101] As an example, the image analysis module (640) may include an optical character recognition (OCR) module (641). The OCR module (641) can convert specific parts of an image contained in a target image into text. For example, if the target image is a menu image of a specific restaurant (e.g., a hamburger restaurant), the OCR module (641) can identify parts of the image containing the types of food available for order, food descriptions, and / or prices contained in the menu image. The OCR module (641) analyzes the text images contained in the parts of the image and converts them into first text in a first language, and can output information regarding the converted text.

[0102] The context analysis module (650) can output analysis information regarding the user's current state and / or environment based on information provided from at least one component among the ASR module (610), camera driving module (620), sensor module (630), or image analysis module (640). As an example, the context analysis module (650) can determine the user's current context information and / or performable tasks inferred based on text portions included in a target image, image portions other than text, and the device's sensing data. For example, the context analysis module (650) can acquire valid data such as GPS information, location information via sound, or information regarding applications such as schedules. The context analysis module (650) can perform inference regarding the user's current state and / or current environment using the acquired valid data. Based on the inference results, the context analysis module (650) can determine tasks to be provided to linked services following the translation service for multimodal services for the user (e.g., applications that can provide speech functions or navigation functions). The context analysis module (650) can obtain context information regarding whether the user is traveling, working, or in a market by comprehensively analyzing, for example, text portions included in the target image, image portions other than text, and the device's sensing data. The context analysis module (650) may be able to extract the sensing data provided by the sensor module (630) at a more detailed level based on the text information obtained from the target image and the information obtained through the analysis of the image. For example, if the situation where the user is traveling is predicted based on the sensing data and a menu image is input as the target image, the context analysis module (650) can infer that the user is in a situation where they intend to place an order or intend to ask a question about a specific food item.For example, if a user inputs an image of a product manual at home, the context analysis module (650) can infer that the user is in a situation where they intend to operate the product or make an inquiry regarding the operation.

[0103] The LLM (660) can perform inference operations for a translation service and / or a linked service reflecting the translation results based on data provided from at least one component among the ASR module (610), camera driving module (620), sensor module (630), image analysis module (640), or context analysis module (650). For example, the LLM (660) can translate a first text, which is a first language, provided by the OCR module (641) into a second text, which is a second language. For example, the LLM (660) can provide processed data for a linked service to be provided in conjunction with the translation service by comprehensively considering information regarding the image other than the text included in the target image provided by the image analysis module (640), the first and / or second text, or context information provided by the context analysis module (650). For example, if it is to provide a speech service in conjunction with the translation service, the LLM (660) can provide one or more recommended example sentences to the user for conversation with the other party based on various acquired information. For example, LLM (660) may provide customized example sentences that reflect the user's needs by additionally considering information obtained by performing an interaction with the user.

[0104] The interface output module (670) can determine a user interface for providing a linked service after a translation service based on context information provided by the context analysis module (650), and perform an operation to output it. For example, the interface output module (670) can determine a linked task that the user can additionally perform, such as utterance, in addition to a translation task, based on the user's context information, and output a user interface for performing the determined linked task.

[0105] The TTS module (680) can convert text into an audible signal and output it by considering a sample selected from among the recommended sample sentences provided by the LLM (660), or a user interface determined by the interface output module (670). For example, in a situation where a user wants to order food at a specific restaurant, the TTS module (680) can output a sentence to order a selected menu to the other party in the desired language as an audible signal.

[0106] FIG. 7 is a control flow diagram for performing a multimodal service in an electronic device according to one embodiment (e.g., the electronic device (300) of FIG. 3).

[0107] An electronic device that performs the operation according to FIG. 7 (e.g., the electronic device (300) of FIG. 3) (hereinafter referred to as 'electronic device (300)') may be one of various forms of electronic devices, such as smartphones having various form factors (e.g., smartphones (391) of FIG. 3), tablets (e.g., tablets (392) of FIG. 3), AR devices (e.g., AR devices (100) of FIG. 1), or other similar computing devices (not shown). Smartphones having various form factors may be, for example, a bar-type smartphone (e.g., the bar-type smartphone (391-1) of FIG. 3), a foldable-type smartphone (e.g., the foldable-type smartphone (391-2) of FIG. 3), or a sliderable (e.g., the sliderable (or rollable)-type smartphone (391-3) of FIG. 3).

[0108] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0109] Referring to FIG. 7, the electronic device (300) can translate a first text, which is a first language included in an image, into a second text, which is a second language, in operation 710. The first language may be, for example, a foreign language unfamiliar to the user (e.g., English, Japanese, Arabic). The second language may be, for example, a native language familiar to the user (e.g., Korean).

[0110] For example, an electronic device (300) may acquire an image (e.g., a preview image or a captured image) obtained by a camera (e.g., the image sensor (350) of FIG. 3) as a target image for translation. The electronic device (310) may identify a text image, which is a partial image included in the acquired target image, and convert the identified text image into a first text. The electronic device (300) may output a result of analyzing the text image using, for example, an OCR function (e.g., OCR (641) included in the image analysis module (640) of FIG. 6). Here, the analysis result may be a first text consisting of a font of a specific language (e.g., a first language) that is recognizable by humans. The electronic device (300) may convert (or translate) the first text into a second text consisting of a font of a second language through, for example, a predetermined AI model (e.g., LLM (660) of FIG. 6) capable of providing a translation function. For example, if the user selects English as the first language and Korean as the second language as options for translation, the electronic device (300) can obtain the first text “It is difficult for a product to succeed these days without incorporating AI-related features.” included in the target image. The electronic device (300) can obtain the second text, which translates the first text “It is difficult for a product to succeed these days without incorporating AI-related features.” into “It is difficult for a product to succeed these days without incorporating AI-related features” using a Korean font. The electronic device (300) can display the second text or convert the second text into an image at the location in the target image where the image corresponding to the first text existed, or around that location.For example, if a user selects English as the first language and Korean as the second language as options for translation, the electronic device (300) can convert and translate the words and / or sentences into text and replace the words or sentences or display them together with the words or sentences if there are multiple words and / or sentences in the target image in a font corresponding to the first language. The electronic device (300) can operate to output the first text and / or the second text as an audible signal by means of a predetermined component (e.g., the TTS module of FIG. 6) (680)) in response to a speech request.

[0111] The electronic device (300) can obtain context information regarding the usage environment in operation 720. Here, the context information may be information that can determine the user's current situation (e.g., a situation where one wants to purchase an item or a situation where one wants to move to a specific place). The electronic device (300) can infer the user's needs based on various types of information collected in relation to the user's current situation, and obtain context information based on the inference result. For example, the electronic device (300) can obtain context information regarding the usage environment based on at least one of a target image, a first text, a second text, or sensing data from at least one sensor.

[0112] The electronic device (300) can obtain context information based on the previously acquired image. For example, the electronic device (300) can obtain context information that "the user is located at a hamburger restaurant" based on the image of a target object (e.g., an object such as a hamburger) included in the acquired image. For example, if the acquired image corresponds to the instruction manual of a specific product, the electronic device (300) can obtain context information that "the user wants to try using the product" and / or "the user wants to ask a question about the product." As an example, the electronic device (300) can obtain context information based on a first text and / or a second text. For example, the electronic device (300) can obtain context information that "the user wants to select a hamburger and / or a drink to order" based on information regarding "the type and price of the hamburger and / or the type and price of the drink" included in the first text included in the acquired image and / or the second text translated from the first text. For example, the electronic device (300) can obtain context information that 'the user intends to order a selected hamburger and / or drink' based on an image of a target object (e.g., an object such as a hamburger) included in an acquired image, a first text, and / or a second text. For example, the electronic device (300) can obtain context information based on sensing data sensed by at least one sensor. The sensing data may include location information sensed by a location sensor, such as a GPS, for example. For example, the electronic device (300) can analyze sound collected using a microphone capable of converting an audible signal into an electrical signal and obtain context information based on the analysis result. For example, the electronic device (300) can obtain context information based on information managed by at least one application.For example, if an application managing personal schedules includes an overseas travel schedule on a given date, the electronic device (300) can obtain context information based on information about the overseas travel (e.g., country, city).

[0113] According to one example, the electronic device (300) can display category items on a display to infer recommended text based on acquired context information. The electronic device (300) can recognize at least one category item selected by the user among the category items. The electronic device (300) can determine target context information corresponding to at least one category item selected by the user from the context information. The electronic device (300) can acquire one or more recommended example sentences based on the determined target context information and output them.

[0114] The electronic device (300) can extract one or more recommended example sentences related to the second text based on context information in operation 730. As an example, the electronic device (300) can infer one or more recommended example sentences based on information analyzing the target image and / or context information. For example, if the electronic device (300) recognizes that the user is traveling abroad based on context information and recognizes that the user is looking at a menu at a restaurant based on the target image, it can infer at least one recommended text, such as recommended text for ordering a desired menu or recommended text for a question about a specific menu.

[0115] According to one example, the electronic device (300) may perform an analysis on a target image to obtain information to consider for inferring recommended examples. The target image to be analyzed by the electronic device (300) may be, for example, an image obtained in operation 710. The target image to be analyzed by the electronic device (300) may be, for example, an image obtained by a camera (e.g., image sensor (350) of FIG. 3) (e.g., a preview image or a captured image). The electronic device (300) may, for example, analyze one or more objects included in the target image to obtain information for inferring recommended examples. The electronic device (300) may, for example, use an OCR function (e.g., OCR (641) included in the image analysis module (640) of FIG. 6) to analyze a text image included in the target image to obtain text information for inferring recommended examples. Here, the text information may consist of a font of a specific language (e.g., a first language) that is recognizable by humans. The electronic device (300) can convert text information composed of a first language font into text information composed of a second language font through, for example, a predetermined AI model (e.g., LLM (660) of FIG. 6).

[0116] The electronic device (300) can perform an interaction with a user in operation 740 to extract one or more processed example sentences. Here, the interaction with the user may be performed by a method such as gestures or voice, or a combination of at least two heterogeneous methods, in addition to basic methods such as key input. As an example, the electronic device (300) can generate example sentences that are processed or modified from recommended example sentences based on information regarding the predicted user's requirements, i.e., the user's intentions, by performing an interaction with the user. Here, the processed recommended example sentence may be a user-customized example sentence that reflects the user's needs relatively compared to the recommended example sentence.

[0117] The electronic device (300) may display one or more recommended examples and / or one or more processed examples in at least one of a first language and / or a second language on a display. For example, the electronic device (300) may display one or more recommended examples related to a second text in at least one of a first language or a second language on a display. For example, the electronic device (300) may display the second text overlaid on the first text or around the first text.

[0118] The electronic device (300) can output an example selected from one or more recommended examples and / or one or more processed examples in operation 750 as an audible signal of the first language.

[0119] FIGS. 8a to 8h are drawings for explaining how to provide a multimodal service in conjunction with a translation result in an electronic device according to one embodiment (e.g., the electronic device (300) of FIG. 3). FIG. 9 is a control flow diagram for performing a multimodal service in conjunction with a translation result in an electronic device according to one embodiment (e.g., the electronic device (300) of FIG. 3).

[0120] In FIGS. 8a through 8h, an AR glass (e.g., the AR device (100) of FIG. 1) is assumed to be an electronic device providing multimodal services (e.g., the electronic device (300) of FIG. 3) (hereinafter referred to as 'electronic device (300)'), but this is merely illustrative, and the operation according to FIG. 9 can be performed in the same way by various types of electronic devices such as smartphones having various form factors (e.g., the smartphones (391) of FIG. 3), tablets (392), or other similar computing devices (not shown). Smartphones having various form factors may be, for example, one of a bar-type smartphone, a foldable-type smartphone, or a sliderable (or rollable)-type smartphone.

[0121] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0122] Referring to FIGS. 8a through 8h or FIG. 9, the electronic device (300) can recognize that a translation function by a user is executed in operation 901. The electronic device (300) can determine whether the translation function is activated based on user input. For example, the electronic device (300) can activate the translation function in response to the operation of the corresponding function button by the user. For example, the electronic device (300) can activate the translation function in response to a voice command from the user. For example, the user can request the electronic device (300) to perform a translation of the subject to be translated by confirming the subject to be translated, pointing the camera at the subject to be translated, and pressing the corresponding function button. For example, the user can request the electronic device (300) to perform a translation of the subject to be translated by confirming the subject to be translated, pointing the camera at the subject to be translated, and saying, “Show me the result of translating this.” For example, the electronic device (300) may activate a translation function when it analyzes a target image and determines that translation is required. In this case, the electronic device (300) may not perform operation 901.

[0123] The electronic device (300) can analyze an image for translation in operation 902. As an example, the electronic device (300) can obtain a preview image by activating a camera function or an execution screen as a specific application is executed. At this time, the execution screen or the preview image may be a target image for translation. Since the operation of the electronic device (300) analyzing an image for translation can be borrowed from the description previously made with reference to FIG. 7, a detailed description has been omitted (see description regarding operation 710 or operation 720 of FIG. 7).

[0124] The electronic device (300) can determine whether text exists in the target image in operation 903. For example, the electronic device (300) can determine whether a partial image corresponding to text of a first language (e.g., English, Japanese, Arabic) is included in the target image. In the following description, an unfamiliar foreign language, rather than the native language familiar to the user, will be referred to as the 'first language,' and the native language familiar to the user will be referred to as the 'second language.' For example, the electronic device (300) may determine the second language based on the content of the utterance when the translation function is executed in response to the user's utterance. For example, if the command uttered by the user is "Translate into Korean," the electronic device (300) may determine the second language as "Korean." For example, the electronic device (300) may determine the second language (e.g., Korean) based on language information according to the default settings. For example, if the command spoken by the user is “translate,” the electronic device (300) can determine “Korean” as the second language, which is set as the default language information.

[0125] FIG. 8a is a diagram illustrating a scenario in which a user (801) uses an electronic device (300) (e.g., AR glasses (or a device with a camera) (803)) to obtain an image corresponding to a menu board with information about the menu displayed in English, which is the first language (e.g., an operation according to operation 901 to operation 903 of FIG. 9).

[0126] In FIG. 8a, a situation is assumed where a user (801) enters a restaurant to eat a hamburger while traveling and looks at the menu. In this assumed situation, the user (801) may wish for the menu, which is written in a first language (e.g., English), to be provided in a second language, such as their native language (e.g., Korean), which they are familiar with. Reflecting this need of the user (801), the AR glasses (803) can activate a translation function.

[0127] For example, a user (801) may look at a target object that needs translation while wearing AR glasses (803). In this case, the AR glasses (803) may acquire a target image (e.g., a preview image or a captured image) (810) containing the target object by means of a mounted image sensor. The AR glasses (803) may display the acquired first image (810) on a display. The AR glasses (803) may recognize that the acquired first image (810) contains a part image (813) that needs translation. Here, the part image that needs translation may be an image corresponding to a first text written in a first language (e.g., English). For example, the first image (810) may contain a food image (811) corresponding to a menu (e.g., an image of a hamburger, a side dish, and / or beverages). For example, the first image (810) may include a partial image (813) in which information about the menu (e.g., types of hamburgers, sides, or drinks, and price information) is displayed in a first language (e.g., English). As an example, AR glasses (803) can acquire text contained in the acquired first image (810) using an OCR function. When text is acquired from the first image (810), AR glasses (803) can identify whether the acquired text is displayed in a first language (e.g., English) or in a second language (e.g., Korean). If the acquired text is displayed in a first language, AR glasses (803) can determine that translation is required.

[0128] The electronic device (300) may not perform an operation for translation if the target image does not contain a partial image corresponding to the first text written in the first language. If the target image contains a partial image corresponding to the first text, the electronic device (300) may perform a translation of the partial image in operation 904. Here, the partial image may be an image containing the first text written in the first language (e.g., English) (see FIG. 8a). For example, the electronic device (300) may obtain the first text from the partial image using an OCR function. The electronic device (300) may generate a second text by translating words and / or sentences included in the obtained first text into a second language. The electronic device (300) may convert the generated second text into a partial image. The electronic device (300) may add the converted partial image to the target image or replace the partial image prior to translation with the converted partial image to display it. For example, the electronic device (300) can display the translated partial image by overlaying it on the pre-translation partial image included in the target image. For example, the electronic device (300) can display the translated partial image around the pre-translation partial image included in the target image.

[0129] FIG. 8b is a diagram illustrating a scenario in which a user (801) uses AR glasses (803) to provide an image corresponding to a menu board that translates information about a menu into a second language, Korean, using a first language, English (e.g., an action according to action 904 of FIG. 9).

[0130] Referring to FIG. 8b, AR glasses (803) can obtain a first text from a partial image (e.g., partial image (813) of FIG. 8a) written in a first language from a target image for translation (e.g., first image (810) of FIG. 8a). For example, AR glasses (803) can use an OCR function to obtain the first text from the partial image (813). AR glasses (803) can obtain a second text written in a second language by performing a translation of the first text obtained based on LLM. AR glasses (803) can convert the obtained second text into a partial image (823) to add it anew, or display a second image (820) that replaces the partial image (813) prior to translation. For example, the second image (820) may include food images (811) corresponding to the menu (e.g., images of a hamburger, a side dish, and / or beverages). For example, the second image (820) may include a partial image (823) in which information about the menu (e.g., types and prices of hamburgers, sides, or beverages) is written in a second language (e.g., Korean).

[0131] The electronic device (300) can obtain context information in operation 905. Here, the context information may be information that can determine the user's current situation (e.g., a situation where one wants to purchase an item or a situation where one wants to move to a specific place). The electronic device (300) can infer the user's needs based on information collected in relation to the user's current situation and obtain context information based on the result of the inference. For example, the electronic device (300) can obtain context information based on a target image (e.g., the first image (810) of FIG. 8a). For example, the electronic device (300) can obtain context information that 'the user is located at a hamburger restaurant' based on an image of a target object (e.g., an object such as a hamburger) included in the first image (810). For example, if the first image (810) corresponds to a manual for a specific product, the electronic device (300) can obtain context information that 'the user wants to try using the product' and / or 'the user wants to ask a question about the product'. For example, the electronic device (300) can obtain context information based on a first text and / or a second text. For example, the electronic device (300) can obtain context information that 'the user wants to select a hamburger and / or a drink to order' based on information regarding "type and price of a hamburger and / or type and price of a drink" included in the first text and / or a second text translated from the first text included in the first image (810). For example, the electronic device (300) can obtain context information that 'the user wants to order a selected hamburger and / or a drink' based on an image of a target object (e.g., an object such as a hamburger) included in the first image (810), the first text, and / or the second text. For example, the electronic device (300) can obtain context information based on sensing data sensed by at least one sensor.Sensing data may include location information sensed by a location sensor, such as GPS, for example. For example, an electronic device (300) may analyze sound collected using a microphone capable of converting an audible signal into an electrical signal, and obtain context information based on the analysis results. For example, the electronic device (300) may obtain context information based on information managed by at least one application. For example, if an application managing personal schedules includes an overseas travel schedule on a given date, the electronic device (300) may obtain context information based on information regarding the overseas travel (e.g., country, city).

[0132] The electronic device (300) can infer one or more recommended texts based on information and / or context information obtained by analyzing a target image (e.g., the first image (810) of FIG. 8a) in operation 906. As an example, the electronic device (300) can infer at least one recommended text, such as recommended text for ordering a desired menu or recommended text for asking a question about a specific menu, when it recognizes based on context information that the user is traveling abroad and recognizes based on the target image that the user is looking at a menu in a restaurant.

[0133] FIG. 8c is a diagram illustrating a scenario in which a user (801) receives recommended text to use a linked service by using translation results from AR glasses (803) (e.g., action 905 or action 906 according to FIG. 9).

[0134] Referring to FIG. 8c, AR glasses (803) can infer information regarding one or more contexts reflecting the needs of a user (801) based on text information obtained from a target image (e.g., the first image (810) of FIG. 8a) (e.g., text information obtained from a partial image (813) of FIG. 8s), image information (e.g., food image (811) included in the first image (810) of FIG. 8), sensing information obtained by at least one sensor, and / or information obtained from a specific application. For example, AR glasses (803) can recognize a situation where the user (801) has visited an overseas restaurant using the user's schedule information and / or GPS information. For example, AR glasses (803) can infer that the user (801) is in a situation where they want to order at a restaurant selling hamburgers based on text information and / or image information included in the target image (810). In this case, the AR glasses (803) can infer one or more recommended texts that reflect the needs of the user (801) by considering the perceived situation. The AR glasses (803) can output one or more inferred recommended texts to the display (831). For example, the AR glasses (803) can display recommended texts in a second language such as “What are the ingredients of this food?”, “Can I know the calories?”, or “Is there a discount if I order more?”. As an example, the AR glasses (803) can assign different priorities to the recommended texts by considering frequency or importance through the user's (801) personal data or history data. As an example, the AR glasses (803) can select and display recommended texts related to the individual's specific situation among the recommended texts by considering frequency or importance through the user's (801) personal data or history data.For example, if the user has a nut allergy, the AR glasses (803) can provide the user (801) with a recommendation text that says, “Does the food you ordered contain nut?”

[0135] FIG. 8d is a diagram illustrating a scenario in which a user (801) selects recommended text to use a linked service by using translation results from AR glasses (803).

[0136] Referring to FIG. 8d, a user (801) can select one of the recommended texts (831) provided by AR glasses (803). For example, the user (801) can select a specific recommended text from the displayed recommended texts (831) by using a finger (841) in virtual space. For example, the user (801) can select “Is there a discount if I order an additional item?” from the displayed recommended texts (831) such as “What are the ingredients of this food?”, “Can I know the calories?”, or “Is there a discount if I order an additional item?”. In this case, the AR glasses (803) can recognize that the recommended text “Is there a discount if I order an additional item?” has been selected by the user (801).

[0137] FIG. 8e is a diagram illustrating a scenario in which a user (801) uses a linked service based on a translation result on AR glasses (803).

[0138] Referring to FIG. 8e, AR glasses (803) can provide a linked service using one recommended text selected by the user (801) from among the provided recommended texts (831). For example, if the AR glasses (803) select a recommended text in a second language, such as “What are the ingredients of this food?”, the user (801) can display text (851) written in a second language (e.g., Korean) corresponding to the recommended text, and text (853) written in a first language (e.g., English) (e.g., What are the ingredients in this food?). AR glasses (803) can display an identifier (855) so that the user (801) can select a speech service as the linked service. If the user (801) wishes to output the corresponding recommended text (853) in the first language as an audible signal, the user (801) can perform an interaction by touching the identifier (855) using a finger (857) in a virtual space. AR glasses (803) can speak ‘What are the ingredients in this food?’ when they recognize an interaction in which an identifier (855) is selected by the user (801).

[0139] As described above, the AR glasses (803) can provide a speech service that inquires about questions as a linked service reflecting the user's (801) needs regarding the translation results. The speech for the recommended text corresponding to the linked service can provide a service where the AR glasses (803) asks a clerk in an unfamiliar language on behalf of the user (801). That is, the AR glasses (803) can not only provide the user (801) with translation results, but also provide a linked service (e.g., speech service) using those results.

[0140] The electronic device (300) can perform an interaction with the user in operation 907. Here, the interaction with the user may be performed by a method such as gestures or voice, or a combination of at least two heterogeneous methods, in addition to basic methods such as key input. The electronic device (300) can perform an interaction with the user in operation 908 to generate custom text that is more user-customized by processing the recommended text to reflect the predicted user requirements. The electronic device (300) can output the custom text for the user in operation 909.

[0141] FIG. 8f is a diagram illustrating a scenario in which a user (801) generates custom text to use a linked service on AR glasses (803) (e.g., actions according to actions 907, 908, and 909 of FIG. 9).

[0142] Referring to FIG. 8f, AR glasses (803) can perform a selection action (action 907) to collect the user's (801) requirements for providing recommended text. For example, AR glasses (803) can display categories of recommended text (e.g., order creation (861), food query (863), place query (865)) that can be selected by the user (801) based on a target image (811) and / or context information. The user (801) can select one of the categories (861, 863, 865) suggested by the AR glasses (803). For example, the user (801) can select a specific category from the displayed categories (861, 863, 865) using a finger (867) in virtual space. For example, the user (801) can select “Order Creation (861)” from among the displayed categories (861, 863, 865), “Food Query (863)”, or “Place Query (865)”.

[0143] FIG. 8g is a diagram illustrating a scenario in which a user (801) provides custom text for an order to a linked service from AR glasses (803) (e.g., actions according to actions 907, 908, and 909 of FIG. 9).

[0144] Referring to FIG. 8g, AR glasses (303) can perform an interaction with the user (801) to place an order when the category “Create Order (861)” is selected by the user (801). For example, AR glasses (303) can display custom text (871) in a second language (e.g., Korean) that says, “Would you like to create a sentence to place an order? Please select the food you would like to eat from the menu.” The user (801) can select a menu to order in a virtual space in response to the custom text (871). For example, the user (801) can perform an interaction in the virtual space by using a finger (873) to select an item corresponding to a specific hamburger (e.g., Double Cheese Burger) from the menu displayed in the second language (see ①). For example, the user (801) can perform an interaction in which they use their finger (877) in virtual space to select an item corresponding to a specific drink (e.g., Zero Cola) from a menu displayed in a second language (see ②). In response, the AR glasses (803) can indicate that the Double Cheeseburger (875) and Zero Cola (879) have been selected by the user (801) as the menu items to order.

[0145] FIG. 8h is a diagram illustrating a scenario to help a user (801) order a selected menu in AR glasses (803) (e.g., actions according to actions 907, 908, and 909 of FIG. 9).

[0146] Referring to FIG. 8h, AR glasses (803) can display recommendation text for ordering a double cheeseburger and a zero coke selected by the user (801) in a first language (e.g., English) and / or a second language (e.g., Korean). For example, AR glasses (803) can display “Shall I order a double cheeseburger and a zero coke?” as recommendation text (881) in the second language. For example, AR glasses (803) can display “I’ll order double cheeseburger and a zero coke.” as recommendation text (883) in the first language. AR glasses (803) can display an identifier (885) so that the user (801) can select a speech service as a linked service. If the user (801) wishes to output the corresponding recommendation text (883) in the first language as an audible signal, the user (801) can perform an interaction of touching the identifier (885) with a finger in a virtual space. AR glasses (803) can speak “I’ll order double cheeseburger and a zero coke.” when they recognize an interaction in which an identifier (885) is selected by the user (801).

[0147] FIGS. 10a to 10d are drawings for illustrating an exemplary provision of a navigation service for a multimodal service linked to a translation result in an electronic device according to one embodiment (e.g., the electronic device (300) of FIG. 3). FIG. 11 is a control flow diagram for performing navigation based on a translation result in a multimodal service in an electronic device according to one embodiment (e.g., the electronic device (300) of FIG. 3).

[0148] In FIGS. 10a to 10d, a bar-type smartphone is assumed to be the electronic device providing multimodal services (e.g., the electronic device (300) of FIG. 3) (hereinafter referred to as 'electronic device (300)'), but this is merely illustrative, and the operation according to FIG. 11 can be performed in the same way by various types of electronic devices such as smartphones having form factors (e.g., the smartphones (391) of FIG. 3), AR glasses (e.g., the AR device (100) of FIG. 1), a tablet (392), or other similar computing devices (not shown). Smartphones having various form factors may be, for example, one of a bar-type smartphone, a foldable-type smartphone, or a sliderable (or rollable)-type smartphone.

[0149] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0150] Referring to FIGS. 10a through 10d or FIG. 11, the electronic device (300) may display a first map written in a first language in operation 1101 (e.g., see FIG. 10a). The first language may be, for example, an unfamiliar foreign language (e.g., English, Japanese, Arabic) rather than a second language, which is a native language familiar to the user. The first map may be provided by, for example, an application that does not support the second language. The first map may be an image captured using a camera of a paper map written in the first language, such as a tourist map.

[0151] FIG. 10a is a drawing for illustrating an example in which a user (1001) uses an electronic device (1003) to view a map image (1010) displayed in a first language (e.g., Japanese).

[0152] Referring to FIG. 10a, an electronic device (1003) can capture a map (1004) written in a first language (e.g., Japanese) and output the captured image (1010) through a screen. For example, a user can take a picture by operating the electronic device (1003) while holding a paper map (1004) in their right hand, so that a specific location (e.g., ssuki no eki) (1011) included in the specific paper map (1004) is included. The electronic device (1003) can display the captured picture as a first screen (1010). On the first screen (1010) displayed on the electronic device (1003), the specific location (e.g., ssuki no eki) (1011) can be displayed in Japanese on the screen. If the user (1001) is not familiar with Japanese, it may not be easy to recognize a specific location (e.g., すすきのえき) (1011) on the map (1010) displayed on the first screen (1010) of the electronic device (1003).

[0153] The electronic device (300) can determine whether a translation request occurs in operation 1102. The electronic device (300) can determine whether a translation request occurs based on user input. For example, the electronic device (300) can activate a translation function in response to operation of a corresponding function button by the user. For example, the electronic device (300) can activate a translation function in response to a voice command from the user. For example, the user can confirm the subject to be translated on the first screen (1010) and request the electronic device (300) to perform a translation of the subject to be translated by pressing the corresponding function button. For example, the user can confirm the subject to be translated on the first screen (1010) and request the electronic device (300) to perform a translation of the subject to be translated by saying, “Show me the result of translating this.”

[0154] When a translation request occurs, the electronic device (300) may, in operation 1103, display a second map in which text included in a first map supporting a first language is converted into a second language (e.g., see FIG. 10b). The second language may be, for example, a native language familiar to the user (e.g., Korean). For example, the electronic device (300) may use an OCR function to obtain a first text in the first language from a partial image marked in the first language on the first map, and translate the obtained first text into a second text in the second language. The electronic device (300) may display the translated second text at the location on the first map where it was marked in the first language.

[0155] FIG. 10b is a drawing for illustrating an example in which a user (1001) uses an electronic device (1003) to check a map image (1020) translated into a second language (e.g., Korean).

[0156] Referring to FIG. 10b, the electronic device (1003) can display a second screen (1020) in which the first text included in the first screen (1010), which was provided in a first language (e.g., Japanese), is translated into a second language (e.g., Korean). For example, the second screen (1020) can show that Susukino Station (1021), which corresponds to the destination the user (1001) wants to go to, is displayed in the second language (e.g., Korean). The user (1001) can easily find Susukino Station (1021), which corresponds to the destination indicated in the second language, on the second screen (1020).

[0157] The electronic device (300) may, in response to an attempt to apply a specific location of the second map to a specific task in operation 1104, provide a linked service utilizing the specific location in the said task (Fig. 10c). For example, the electronic device (300) may display on the display an identifier corresponding to one or more linked services available to the user based on the translation result (e.g., identifiers of Fig. 10c (1035a, 1035b, 1035c)). For example, if an interaction is made by the user to apply a specific location of the second map to a specific application that provides directions, the electronic device (300) may provide directions from the user's current location to the specific location.

[0158] FIG. 10c is a diagram illustrating the use of a linked service that guides a user (1001) to a specific location (1031) included in a map image (1020) translated into a second language (e.g., Korean) using an electronic device (1003).

[0159] Referring to FIG. 10c, the electronic device (1003) can recognize that the user (1001) performs an interaction in which, after selecting Susukino Station (1021), which corresponds to the destination, with a finger (1033) on a third screen (1030) displayed in a second language (e.g., Korean), the user moves to the identifier (1035a) of a specific application (e.g., a navigation application) that provides the linked service among the identifiers (1035a, 1035b, 1035c) corresponding to multiple applications that can use the linked service (①). When the electronic device (1003) recognizes the interaction by the user (1001), it can recognize that navigation from the user's (1001) current location to the destination Susukino Station (1021) has been requested.

[0160] The electronic device (300) can, in operation 1105, translate text in a second language (e.g., Korean) corresponding to a specific location selected by the user in the second map into a first language (e.g., Japanese), and apply the second text translated into the first language (e.g., Japanese) to a selected specific task to perform an operation for a linked service (see FIG. 10d).

[0161] FIG. 10d is a diagram illustrating an example in which a user (1001) uses an electronic device (1003) to use a navigation service, which is a linked service, based on a translation result.

[0162] Referring to FIG. 10d, when the electronic device (1003) recognizes an interaction with a user (1001) requesting directions from the current location (1041) to the destination (1043), it can display a route (1045) from the current location (1041) to the destination (1043) on a fourth screen (1040) that runs a directions application supporting a first language (e.g., Japanese).

[0163] FIG. 12 is an exemplary block diagram of an electronic device (1201) (e.g., the electronic device (300) of FIG. 3) in a network environment (1200) according to various embodiments.

[0164] Referring to FIG. 12, in a network environment (1200), an electronic device (1201) may communicate with an electronic device (1202) through a first network (1298) (e.g., a short-range wireless communication network) or with at least one of an electronic device (1204) or a server (1208) through a second network (1299) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1201) may communicate with the electronic device (1204) through a server (1208). According to one embodiment, the electronic device (1201) may include a processor (1220), memory (1230), input module (1250), sound output module (1255), display module (1260), audio module (1270), sensor module (1276), interface (1277), connection terminal (1278), haptic module (1279), camera module (1280), power management module (1288), battery (1289), communication module (1290), subscriber identification module (1296), or antenna module (1297). In some embodiments, at least one of these components (e.g., connection terminal (1278)) may be omitted from the electronic device (1201), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1276), camera module (1280), or antenna module (1297)) may be integrated into a single component (e.g., display module (1260)).

[0165] The processor (1220) can, for example, execute software (e.g., program (1240)) to control at least one other component (e.g., hardware or software component) of the electronic device (1201) connected to the processor (1220) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1220) can store commands or data received from other components (e.g., sensor module (1276) or communication module (1290)) in volatile memory (1232), process the commands or data stored in volatile memory (1232), and store the resulting data in non-volatile memory (1234). According to one embodiment, the processor (1220) may include a main processor (1221) (e.g., a central processing unit or an application processor) or an auxiliary processor (1223) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1201) includes a main processor (1221) and an auxiliary processor (1223), the auxiliary processor (1223) may be configured to use less power than the main processor (1221) or to be specialized for a designated function. The auxiliary processor (1223) may be implemented separately from the main processor (1221) or as part thereof.

[0166] The auxiliary processor (1223) may control at least some of the functions or states associated with at least one component of the electronic device (1201) (e.g., display module (1260), sensor module (1276), or communication module (1290)) on behalf of the main processor (1221) while the main processor (1221) is in an inactive (e.g., sleep) state, or together with the main processor (1221) while the main processor (1221) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1223) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1280) or communication module (1290)). According to one embodiment, the auxiliary processor (1223) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1201) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1208)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0167] The memory (1230) can store various data used by at least one component of the electronic device (1201) (e.g., processor (1220) or sensor module (1276)). The data may include, for example, software (e.g., program (1240)) and input data or output data for related commands. The memory (1230) may include volatile memory (1232) or non-volatile memory (1234).

[0168] The program (1240) may be stored as software in memory (1230) and may include, for example, an operating system (1242), middleware (1244), or an application (1246).

[0169] The input module (1250) can receive commands or data to be used for a component of the electronic device (1201) (e.g., processor (1220)) from outside the electronic device (1201) (e.g., user). The input module (1250) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0170] The sound output module (1255) can output a sound signal to the outside of the electronic device (1201). The sound output module (1255) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0171] The display module (1260) can visually provide information to an external (e.g., user) of the electronic device (1201). The display module (1260) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1260) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0172] The audio module (1270) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1270) can acquire sound through an input module (1250) or output sound through an audio output module (1255) or an external electronic device (e.g., electronic device (1202)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1201).

[0173] The sensor module (1276) can detect the operating state of the electronic device (1201) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1276) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0174] The interface (1277) may support one or more specified protocols that can be used for the electronic device (1201) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1202)). According to one embodiment, the interface (1277) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0175] The connection terminal (1278) may include a connector through which the electronic device (1201) can be physically connected to an external electronic device (e.g., electronic device (1202)). According to one embodiment, the connection terminal (1278) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0176] The haptic module (1279) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (1279) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0177] The camera module (1280) can capture still images and video. According to one embodiment, the camera module (1280) may include one or more lenses, image sensors, image signal processors, or flashes.

[0178] The power management module (1288) can manage power supplied to the electronic device (1201). According to one embodiment, the power management module (1288) may be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0179] The battery (1289) can supply power to at least one component of the electronic device (1201). According to one embodiment, the battery (1289) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0180] The communication module (1290) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1201) and an external electronic device (e.g., electronic device (1202), electronic device (1204), or server (1208)), and the performance of communication through the established communication channel. The communication module (1290) may include one or more communication processors that operate independently of the processor (1220) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1290) may include a wireless communication module (1292) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1294) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1204) through a first network (1298) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1299) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1292) can identify or authenticate the electronic device (1201) within a communication network such as the first network (1298) or the second network (1299) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1296).

[0181] The wireless communication module (1292) can support 5G networks and next-generation communication technologies following 4G networks, such as new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1292) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1292) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1292) can support various requirements specified in the electronic device (1201), external electronic device (e.g., electronic device (1204)), or network system (e.g., second network (1299)). According to one embodiment, the wireless communication module (1292) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0182] An antenna module (1297) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1297) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1297) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1298) or a second network (1299), may be selected from the plurality of antennas, for example, by a communication module (1290). A signal or power may be transmitted or received between the communication module (1290) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1297).

[0183] According to various embodiments, the antenna module (1297) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0184] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0185] According to one embodiment, commands or data may be transmitted or received between the electronic device (1201) and an external electronic device (1204) through a server (1208) connected to a second network (1299). Each of the external electronic devices (1202, or 1204) may be the same or a different type of device as the electronic device (1201). According to one embodiment, all or part of the operations performed on the electronic device (1201) may be performed on one or more of the external electronic devices (1202, 1204, or 1208). For example, if the electronic device (1201) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1201) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1201). The electronic device (1201) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1201) may provide ultra-low latency services, for example, using distributed computing or mobile edge computing. In one embodiment, the external electronic device (1204) may include an Internet of Things (IoT) device. The server (1208) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1204) or server (1208) may be included within the second network (1299). The electronic device (1201) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0186] Although not included in the scenarios described above, if a user, while wearing AR glasses and receiving a real-time surrounding image generation service based on a cloud environment, enters a place with poor network communication conditions, such as an underground area, they may be able to receive a seamless service by utilizing the model switching method proposed in this document.

[0187] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0188] As an example, the electronic device (300) may include at least one sensor including an image sensor (350). The electronic device (300) may include a memory (320) including one or more storage media for storing instructions. The electronic device (300) may include at least one processor (310) including a processing circuit. When the instructions are executed individually or collectively by the at least one processor (310), the electronic device (300) may be caused to perform at least one operation. The at least one operation may include an operation of obtaining a first text of a first language from a target image output by the image sensor (350). The at least one operation may include an operation of obtaining a second text by translating the obtained first text into a second language. The at least one operation may include an operation of obtaining context information regarding a usage environment based on at least one of the target image, the first text, the second text, or the sensing data of the at least one sensor. The at least one operation may include an operation of outputting one or more recommended texts related to the obtained second text based on the obtained context information.

[0189] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the action of selecting one of the one or a number of recommended texts.

[0190] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the operation of outputting the selected recommended text as an audible signal of the first language.

[0191] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform an action of: performing an interaction with a user to obtain information regarding the user's intention.

[0192] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform an operation of: modifying at least one of the one or a number of recommended texts based on information regarding the acquired user intention and outputting a processed recommended text.

[0193] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the operation of displaying the processed recommendation text on a display in at least one of the first language or the second language.

[0194] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the operation of outputting the processed recommended text as an audible signal of the first language in response to a speech request.

[0195] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the operation of displaying one or more recommended texts related to the acquired second text on a display (340) in at least one of the first language or the second language.

[0196] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the operation of: overlapping the second text with the first text or displaying it around the first text.

[0197] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the action of displaying category items to infer recommended text based on the acquired context information.

[0198] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform an operation of determining target context information corresponding to one category item selected from the indicated category items from the acquired context information.

[0199] As an example, when the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) may be caused to perform the operation of outputting the one or more recommended texts based on the determined target context information.

[0200] As an example, the operation method of the electronic device (300) may include the operation of obtaining a first text of a first language from a target image output by an image sensor (350). The operation method may include the operation of obtaining a second text that translates the obtained first text into a second language. The operation method may include the operation of obtaining context information regarding a usage environment based on at least one of the target image, the first text, the second text, or the sensing data of at least one sensor. The operation method may include the operation of outputting one or more recommended texts related to the obtained second text based on the obtained context information.

[0201] As an example, the above operation method may include the operation of selecting one of the one or more recommended texts.

[0202] As an example, the above operation method may include the operation of outputting the selected recommended text as an audible signal of the first language.

[0203] As an example, the above method of operation may include an operation of performing an interaction with a user to obtain information regarding the user's intention.

[0204] As an example, the above operation method may include an operation of modifying at least one of the one or a plurality of recommended texts based on information regarding the acquired user intention and outputting a processed recommended text.

[0205] As an example, the above operation method may include an operation of displaying the processed recommendation text in at least one of the first language or the second language.

[0206] As an example, the above operation method may include an operation of outputting the processed recommended text as an audible signal of the first language in response to a speech request.

[0207] As an example, the above operation method may include an operation of displaying one or more recommended texts related to the acquired second text in at least one of the first language or the second language.

[0208] As an example, the above method of operation may include the operation of overlapping the second text with the first text or displaying it around the first text.

[0209] As an example, the operation of outputting one or more recommended texts may include the operation of displaying category items to infer recommended texts based on the acquired context information.

[0210] As an example, the operation of outputting one or more recommended texts may include the operation of determining target context information corresponding to one category item selected among the displayed category items from the acquired context information.

[0211] As an example, the operation of outputting one or more recommended texts may include the operation of outputting one or more recommended texts based on the determined target context information.

[0212] As an example, a recording medium may store instructions that can be read by a computer. When executed by at least part of at least one processor (310) included in the electronic device (300), the instructions may cause the electronic device (300) to perform at least one operation. The at least one operation may include an operation of obtaining a first text of a first language from a target image output by an image sensor (350). The at least one operation may include an operation of obtaining a second text that translates the obtained first text into a second language. The at least one operation may include an operation of obtaining context information regarding a usage environment based on at least one of the target image, the first text, the second text, or the sensing data of the at least one sensor. The at least one operation may include an operation of outputting one or more recommended texts related to the obtained second text based on the obtained context information.

[0213] As an example, the above at least one operation may include the operation of selecting one of the above one or a number of recommended texts.

[0214] As an example, the above at least one operation may include an operation of outputting the selected recommended text as an audible signal of the first language.

[0215] As an example, the above at least one operation may include an operation of performing an interaction with a user to obtain information regarding the user's intention.

[0216] As an example, the above at least one operation may include an operation of modifying at least one of the one or more recommended texts based on information regarding the acquired user intention and outputting a processed recommended text.

[0217] As an example, the above at least one operation may include displaying the processed recommendation text in at least one of the first language or the second language.

[0218] As an example, the above at least one operation may include an operation of outputting the processed recommended text as an audible signal of the first language in response to a speech request.

[0219] As an example, the above at least one operation may include an operation of displaying one or more recommended texts related to the acquired second text in at least one of the first language or the second language.

[0220] As an example, the above at least one operation may include the operation of overlapping the second text with the first text or displaying it around the first text.

[0221] As an example, the operation of outputting one or more recommended texts may include the operation of displaying category items to infer recommended texts based on the acquired context information.

[0222] As an example, the operation of outputting one or more recommended texts may include the operation of determining target context information corresponding to one category item selected among the displayed category items from the acquired context information.

[0223] As an example, the operation of outputting one or more recommended texts may include the operation of outputting one or more recommended texts based on the determined target context information.

[0224] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0225] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0226] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0227] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory (940)) readable by a machine (e.g., electronic device (110, 900)). For example, a processor (e.g., processor (910)) of the machine (e.g., electronic device (110, 900)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term is used when data is stored It does not distinguish between cases where data is stored semi-permanently on a medium and cases where it is stored temporarily.

[0228] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0229] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (300), At least one sensor including an image sensor (350); Memory (320) comprising one or more storage media for storing instructions; and It includes at least one processor (310) including a processing circuit, and When the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) is caused to perform at least one operation, and The above at least one operation is, The operation of obtaining a first text of a first language from a target image output by the image sensor (350); The operation of obtaining a second text obtained by translating the first text obtained above into a second language; An operation of obtaining context information regarding a usage environment based on at least one of the above target image, the above first text, the above second text, or the sensing data of the above at least one sensor; and An operation of outputting one or more recommended texts related to the second text obtained above based on the context information obtained above; Electronic device (300) including 2. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) is: The action of selecting one of the above one or more recommended texts; and The operation of outputting the selected recommended text as an audible signal of the first language. An electronic device (300) that causes to perform.

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) is: An action of performing an interaction with a user to obtain information regarding the user's intention; and An operation of outputting a processed recommended text by modifying at least one of the one or more recommended texts based on the information regarding the user intent obtained above. An electronic device (300) that causes to perform.

4. In Paragraph 3, When the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) is: The operation of displaying the above-mentioned processed recommendation text on a display in at least one of the first language or the second language; and The operation of outputting the processed recommended text as an audible signal of the first language in response to a speech request An electronic device (300) that causes to perform.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) is: The operation of displaying one or more recommended texts related to the second text obtained above on the display (340) in at least one of the first language or the second language. An electronic device (300) that causes to perform.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) is: The operation of overlaying the second text on the first text or displaying it around the first text. An electronic device (300) that causes to perform.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by at least one processor (310), the electronic device (300) is: An operation to display category items for inferring recommended text based on the above-mentioned acquired context information; An operation to determine target context information corresponding to one category item selected from among the displayed category items from the context information obtained above; and The operation of outputting one or more recommended texts based on the target context information determined above. An electronic device (300) that causes to perform.

8. In the method of operating the electronic device (300), The operation of obtaining a first text of a first language from a target image output by an image sensor (350); The operation of obtaining a second text obtained by translating the first text obtained above into a second language; An operation of obtaining context information regarding a usage environment based on at least one of the above target image, the above first text, the above second text, or the sensing data of the above at least one sensor; and An operation of outputting one or more recommended texts related to the second text obtained above based on the context information obtained above; A method of operation including 9. In Paragraph 8, The action of selecting one of the above one or more recommended texts; and The operation of outputting the selected recommended text as an audible signal of the first language. A method of operation including 10. In Paragraph 8 or 9, An action of performing an interaction with a user to obtain information regarding the user's intention; and An operation of outputting a processed recommended text by modifying at least one of the one or more recommended texts based on the information regarding the user intent obtained above. A method of operation including 11. In Paragraph 10, The operation of displaying the above-mentioned processed recommendation text in at least one of the first language or the second language; and The operation of outputting the processed recommended text as an audible signal of the first language in response to a speech request A method of operation including 12. In any one of paragraphs 8 through 11, The operation of displaying one or more recommended texts related to the second text obtained above in at least one of the first language or the second language. A method of operation including 13. In any one of paragraphs 8 through 12, The operation of overlaying the second text on the first text or displaying it around the first text. A method of operation including 14. In any one of paragraphs 8 through 13, The operation of outputting one or more of the above recommended texts is, An operation to display category items for inferring recommended text based on the above-mentioned acquired context information; An operation to determine target context information corresponding to one category item selected from among the displayed category items from the context information obtained above; and The operation of outputting one or more recommended texts based on the target context information determined above. A method of operation including 15. In a recording medium storing computer-readable instructions, When the above instructions are executed by at least part of at least one processor (310) included in the electronic device (300), the electronic device (300) causes at least one operation to be performed, and The above at least one operation is: The operation of obtaining a first text of a first language from a target image output by an image sensor (350); The operation of obtaining a second text obtained by translating the first text obtained above into a second language; An operation of obtaining context information regarding a usage environment based on at least one of the above target image, the above first text, the above second text, or the sensing data of the above at least one sensor; and An operation of outputting one or more recommended texts related to the second text obtained above based on the context information obtained above. A recording medium including

Images