Electronic device for performing multitasking by using wearable electronic device, operating method thereof, and storage medium

Abstract

According to an embodiment, a non-transitory storage medium stores computer-readable instructions, the instructions being configured to cause an electronic device to perform one or more operations when executed by at least one processor of the electronic device. The one or more operations may comprise the operations of: connecting to a first external electronic device; transmitting an audio signal related to multimedia content to the first external electronic device while outputting the multimedia content; identifying a call connection with a second external electronic device; generating information corresponding to an utterance of a user of the second external electronic device in response to identifying the call connection while outputting the multimedia content; and displaying the information corresponding to the utterance of the user so as to overlap at least a portion of the multimedia content.

Description

Electronic device for performing multitasking using a wearable electronic device, method of operation thereof, and storage medium

[0001] One embodiment disclosed in this document relates to an electronic device for performing multitasking using a wearable electronic device, a method of operation thereof, and a storage medium.

[0002] Various services and additional functions provided through user terminals, such as electronic devices like smartphones, are gradually increasing. To enhance the utility value of these electronic devices and satisfy the needs of diverse users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer a wide range of functions. Consequently, the various functions provided through these electronic devices are also becoming increasingly sophisticated.

[0003] To perform operations related to connections (e.g., communication links) between the electronic device and external electronic devices (or peripheral electronic devices) (e.g., laptop PC, tablet PC, and / or wearable device), the electronic device may search for external electronic devices, and if the searched external electronic devices exist, it may output information about the searched external electronic devices as a search result.

[0004] In addition, information about various external electronic devices located around the electronic device is displayed as search results, and various services can be utilized through connections between the electronic device and external electronic devices; users may desire to receive more convenient and diverse services depending on their environment or situation.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] According to one embodiment, the electronic device may include a communication circuit including a first communication circuit and a second communication circuit, a display, at least one processor including a processing circuit, and a memory for storing instructions.

[0007] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to connect with a first external electronic device through the first communication circuit.

[0008] According to one embodiment, the instructions may be configured to transmit an audio signal associated with the multimedia content to the first external electronic device through the first communication circuit while outputting the multimedia content through the display, when executed individually or collectively by the at least one processor.

[0009] According to one embodiment, the instructions may be configured to confirm a call connection with a second external electronic device through the second communication circuit when executed individually or collectively by the at least one processor.

[0010] According to one embodiment, the instructions may be configured to generate information corresponding to a user’s utterance of the second external electronic device in response to confirming the call connection while outputting the multimedia content through the display, when executed individually or collectively by the at least one processor.

[0011] According to one embodiment, the instructions may be configured to display information corresponding to the user's utterance through the display so as to be superimposed on at least a part of the multimedia content when executed individually or collectively by the at least one processor.

[0012] According to one embodiment, a method for performing multitasking in an electronic device may include an operation of connecting to a first external electronic device.

[0013] According to one embodiment, the method may include the operation of transmitting an audio signal associated with the multimedia content to the first external electronic device while outputting the multimedia content.

[0014] According to one embodiment, the method may include an operation to confirm a call connection with a second external electronic device.

[0015] According to one embodiment, the method may include an operation of generating information corresponding to a user’s utterance of the second external electronic device in response to confirming the call connection while outputting the multimedia content.

[0016] According to one embodiment, the method may include an operation of displaying information corresponding to the user's utterance so as to be superimposed on at least a part of the multimedia content.

[0017] According to one embodiment, in a non-transient storage medium storing computer-readable instructions, the instructions are configured to cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, wherein the at least one operation may include an operation of connecting to a first external electronic device.

[0018] According to one embodiment, the at least one operation may include transmitting an audio signal associated with the multimedia content to the first external electronic device while outputting the multimedia content.

[0019] According to one embodiment, the at least one operation may include an operation to confirm a call connection with a second external electronic device.

[0020] According to one embodiment, the at least one operation may include an operation of generating information corresponding to a user’s utterance of the second external electronic device in response to confirming the call connection while outputting the multimedia content.

[0021] According to one embodiment, the at least one operation may include an operation of displaying information corresponding to the user's utterance so as to be superimposed on at least a part of the multimedia content.

[0022] FIG. 1a is a block diagram of an electronic device in a network environment according to one embodiment.

[0023] FIG. 1b is an example of a generative artificial intelligence system according to one embodiment.

[0024] FIG. 2 is an internal block diagram of an electronic device according to one embodiment.

[0025] FIG. 3a is a diagram illustrating a method for processing an AI-based audio signal in a multitasking environment according to one embodiment.

[0026] FIG. 3b is a diagram illustrating the structure of a prompt for generating text related to a call according to one embodiment.

[0027] FIG. 3c is a diagram illustrating the structure of a prompt for creating a graphic object related to a call according to one embodiment.

[0028] FIG. 4 is an operation flowchart of an electronic device for performing multitasking using a wearable electronic device according to one embodiment.

[0029] FIG. 5 is a screen example of call content displayed on multimedia content according to one embodiment.

[0030] FIG. 6 is an example screen diagram showing a summary of call content displayed on multimedia content according to one embodiment.

[0031] FIG. 7 is a first operation flowchart of an electronic device when simultaneously performing a call and multimedia content playback according to one embodiment.

[0032] FIG. 8 is a second operation flowchart of an electronic device when simultaneously performing a call and multimedia content playback according to one embodiment.

[0033] FIG. 9 is an example screen diagram showing call content displayed on multimedia content according to one embodiment using a first display method.

[0034] FIG. 10 is an example screen diagram showing call content displayed on multimedia content according to one embodiment using a second display method.

[0035] FIG. 11 is an example of a screen showing call content displayed on multimedia content according to one embodiment using a third display method.

[0036] FIG. 12 is an example screen diagram showing call content displayed on multimedia content according to one embodiment using a fourth display method.

[0037] FIG. 13 is a first screen example diagram for explaining the output volume of multimedia content and call, respectively, according to one embodiment.

[0038] FIG. 14 is a second screen example diagram for explaining the output volume of multimedia content and call, respectively, according to one embodiment.

[0039] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0040] FIG. 1a is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment. Referring to FIG. 1a, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0041] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0042] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0043] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0044] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0045] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0046] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0047] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0048] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0049] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0050] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0051] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0052] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0053] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0054] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0055] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0056] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0057] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0058] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0059] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0060] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0061] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0062] In the following detailed description, reference numbers in the drawings may be assigned identically or omitted for configurations that can be easily understood through prior embodiments, and detailed descriptions thereof may also be omitted. An electronic device (101) according to one embodiment disclosed in this document may be implemented by selectively combining configurations of different embodiments, and a configuration of one embodiment may be replaced by a configuration of another embodiment. For example, it should be noted that the present invention is not limited to specific drawings or embodiments.

[0063] Applications with various functions can be executed simultaneously and operated in multiple ways by the multi-tasking function of the electronic device (101). Accordingly, among the multiple applications running, the execution screen of one application selected by the user is provided through the display, while the remaining applications are operated in the background and the display of their execution screens can be omitted.

[0064] For example, when displaying a screen provided by an application that plays multimedia content based on multitasking, a call connection request may be received while the screen is being displayed. In the event that a situation arises where the audio signal (or audio data) of the multimedia content and the audio signal of the call must be output simultaneously, the electronic device (101) may provide a function that allows the user to set the volume for each. Accordingly, the user may listen to the audio of the multimedia content playback while mainly listening to the other party's voice during the call, or may make a call while mainly listening to the audio of the multimedia content playback. In such a situation where the audio signal of the multimedia content and the audio signal of the call must be output simultaneously, the user can directly adjust the volume of the multimedia content and the call volume, respectively. In addition to the method of adjusting the call volume, the 'answering a call via text' function provides a function that does not output the audio signal during the call and provides a call using text. Volume-related action triggers may include cases where, due to user operation, the call volume becomes lower than the volume for multimedia content, the volume for multimedia content becomes lower than the call volume, or the volume is lowered to a specific volume level (or designated value) or lower.

[0065] When a call is connected while displaying a multimedia content playback screen, the electronic device (101) may stop the playback of the multimedia content and provide a call service, or continue to output the multimedia content screen that was being output and stop only the output of the audio signal of the multimedia content and provide a call service. In addition, if the user wants to use the call service without stopping the playback of the multimedia content and directly adjusts the volume of the multimedia content and the call volume respectively, it may be cumbersome for the user to manually adjust the volume at every moment. For example, when the audio signal of the multimedia content and the audio signal of the call occur simultaneously, there may be moments when the user cannot properly understand the two audio signals because they are output through a single output device, such as earphones. Therefore, if the user cannot hear important call content, they must ask the other party to repeat it, and if a situation occurs where the sound of the multimedia content cannot be heard, they must perform an operation to rewind the playback point, which may be cumbersome for the user.

[0066] For example, a user might increase the volume of multimedia content and decrease the call volume if they determine that the playback of multimedia content is more important than the call service. However, the importance of the call can change every moment depending on the other party and the content of the conversation. Furthermore, even within multimedia content, the importance of the audio may increase during dialogue or scenes that must not be missed. As the importance of multimedia content and the call service changes at every moment, it is difficult to multitask efficiently by simply adjusting the volume manually.

[0067] Furthermore, in the case of the 'Answer Call via Text' feature, since the call content is displayed as text, the keyboard for text input remains active even when a multimedia content playback application is running, which may obscure the multimedia playback screen. Therefore, while the 'Answer Call via Text' feature can provide call services even in situations where voice calls are difficult by delivering text-based call content, it presents difficulties when performing multitasking.

[0068] According to one embodiment, an electronic device for performing multitasking using a wearable electronic device, a method of operation thereof, and a storage medium can be provided so as to check the importance of a call in real time and adjust the volume of the call or display the call content on the multimedia content in a situation where call services and multimedia content playback must be performed simultaneously.

[0069] According to one embodiment, in a situation where call services and multimedia content playback must be performed simultaneously, AI can be used to determine the importance of a call in real time to adjust the volume or allow the user to visually check the call content.

[0070] According to one embodiment, the volume control operation may include an operation to control only the volume of call audio or to control only the volume of audio included in multimedia content. Additionally, the volume control operation may include not only an operation to make the audio inaudible or audible from the user's perspective, but also an operation to make the volume smaller or larger.

[0071] FIG. 1b is an example of a generative artificial intelligence system according to one embodiment.

[0072] Referring to FIG. 1b, the user query / response interface (10), application / service component (30), knowledge repository (20), AI framework (40), and generative AI model (60) may be stored in memory (e.g., memory (130) in FIG. 1a) or on a separate server (108). At least some of the user query / response interface (10), application / service component (30), knowledge repository (20), AI framework (40), or generative AI model (60) may be implemented in software or in hardware.

[0073] According to one embodiment, a user query / response interface (10) may receive user input. User input may be in the form of natural language, images, and / or videos, but is not limited thereto. Additionally, context information may be transmitted along with the user input. Context information may include various additional information at the time of user input. For example, additional information may include information about the application currently being used by the user or the user's location information. Additionally, user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, user input may be in a non-natural language form, such as selecting a menu. The user query / response interface (10) may output results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The user query / response interface (10) may output results of a generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and it can also be provided in a form such as the action requested by the user.

[0074] The AI ​​framework (40) can receive user input and coordinate and control each component necessary to perform the user's intent based on the user's query.

[0075] User input received from the user query / response interface (10) can be transmitted to a prompt design component (41). The prompt design component (41) can be used to generate prompts suitable for inputting user input into a large language model (LLM), a large vision model (LVM), or a large multimodal model (LMM). The prompt design component (41) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. Based on user input, the prompt design component (41) can generate prompts by accessing a knowledge component containing user preference data, a prompt library, and prompt examples, and can transmit the generated prompts to the LLM or LMM.

[0076] The API / Plug-in management component (42) can perform the role of communicating with external information when there is a request for additional information when user input is passed as input to a generative model. The API / Plug-in management component (42) establishes a channel to communicate with the outside of the AI ​​interface through the API, and through the established channel, it can access various data sources (e.g., knowledge repositories (20)). In addition, the API / Plug-in management component (42) can request the application / service component (30) through the API to perform an action that ultimately executes the user input, rather than an intermediate result, in the application or service. The information obtained from the outside may be used to generate a prompt in the prompt design component (41) along with the user input, or it may be passed as input to the generative model.

[0077] The output modification component (or refiner component) (43) can fine-tune the output of the generative model. For example, the output modification component (43) can verify whether the content generated through the LLM and / or LMM is irrelevant, contains biased content, or contains harmful content. Additionally, the output modification component (43) can determine the extent to which the output matches the desired result and, if necessary, proceed with the additional process. Furthermore, the output modification component (43) can configure and provide hints to the user to avoid unwanted output.

[0078] A generative AI model (60) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A generative AI model (60) may include a model that generates images and / or a model that generates language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational auto encoders), and examples include diffusion-based generative models that use VAEs and transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs (large multimodal models) that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0079] FIG. 2 is an internal block diagram of an electronic device according to one embodiment. In describing FIG. 2, reference will be made to FIG. 3a. FIG. 3a is a diagram illustrating a method for processing an AI-based audio signal in a multitasking environment according to one embodiment.

[0080] Referring to FIG. 2, an electronic device (201) (e.g., the electronic device (101) of FIG. 1a) may include a display (260) (e.g., the display module (160) of FIG. 1a), at least one processor (220) (e.g., the processor (120) of FIG. 1a), and a memory (230) (e.g., the memory (130) of FIG. 1a). An electronic device (201) according to one embodiment may further include a communication circuit (290) (e.g., the communication module (190) of FIG. 1a). The at least one processor (220) may include processing circuitry. Not all components shown in FIG. 2 are essential components of the electronic device (201), and the electronic device (201) may be implemented by more or fewer components than those shown in FIG. 2.

[0081] According to one embodiment, the communication circuit (290) can communicate with at least one external electronic device under the control of the processor (220). According to one embodiment, the communication circuit (290) can communicate using at least one communication method among Zigbee, Z-Wave, Wi-Fi, Bluetooth, UWB (ultra-wide band), Wireless USB, and NFC (near field communication). For example, the communication circuit (290) may include a first communication circuit and a second communication circuit. For example, the first communication circuit can establish communication with an external electronic device (e.g., wearable electronic device) (e.g., earphones)) (202) through a first network (e.g., the first network (198) of FIG. 1a). For example, the processor (220) can communicate with an external electronic device (e.g., a wearable electronic device) (202) via Bluetooth legacy. In the following description, a wearable electronic device is used as an example of an external electronic device that outputs audio signals of multimedia content and / or audio signals of a call, but this is merely an example, and it goes without saying that any device capable of listening to audio signals is possible. For example, audio signals can be listened to using an external electronic device such as a speaker through a communication connection with the electronic device (201). Therefore, the external electronic device (202) that is communicated with the electronic device (201) can also be referred to as the first external electronic device, but for convenience of explanation, the case where the external electronic device (202) is a wearable electronic device will be used as an example.

[0082] The second communication circuit can communicate with a counterparty electronic device (e.g., a call partner) through a second network (e.g., the second network (199) of FIG. 2). Here, the counterparty electronic device may be referred to as a second external electronic device.

[0083] According to one embodiment, when an application for playing multimedia content (hereinafter referred to as a multimedia content application) is executed under the control of the processor (220), the display (260) may display a screen corresponding to said application. For example, the display (260) may output (or display) multimedia content. For example, multimedia content may refer to content containing audio signals, such as video, streaming video, or media data. The processor (220) may transmit an audio signal corresponding to the multimedia content application currently being executed to an external electronic device (202) through a communication circuit (290) (or a first communication circuit). Additionally, when an application for making a call is executed, the display (260) may display a screen corresponding to the call application. According to one embodiment, in a multitasking state where the multimedia content application and the call application are executed simultaneously, the processor (220) may not display the call screen itself depending on the importance of the call. For example, the processor (220) may use information related to the call to determine the importance of the call, and if the information related to the call includes spam information, it may not display the call screen or may not connect the call itself.

[0084] According to one embodiment, the processor (220) can transmit an audio signal (e.g., a dial tone or a dial tone) corresponding to a call application to an external electronic device (202) through a communication circuit (290) (or a first communication circuit), and when a call is connected, it can transmit a voice signal corresponding to the other party's speech to the external electronic device (202). According to one embodiment, the processor (220) can simultaneously transmit an audio signal generated by content playback and an audio signal related to a call to the external electronic device (202) in a multitasking state. The processor (220) can mix the audio signal of the content and the audio signal related to the call, and when mixing, it can apply a volume level set for the content and the call, respectively. For example, the mixed audio signal may be (audio signal of the call * call volume level) + (audio signal of the content * content volume level). As described above, the processor (220) can transmit the mixed audio signal in a multitasking state to an external electronic device (202).

[0085] According to one embodiment, the processor (220) can determine the importance (or priority) of a call in a multitasking state in which a multimedia content application and a call application are executed simultaneously. The processor (220) may use information related to the call to determine the importance of the call. For example, the information related to the call may include at least one of a name, a phone number, a relationship with the other party (e.g., family, friend, coworker), the time of the call connection, the volume at the time of the call connection, or the content of the call.

[0086] According to one embodiment, the processor (220) may determine that the priority of multimedia content is higher than the priority of the call application when the importance of the call is low, for example, when the volume of multimedia content is set higher than the call volume. According to one embodiment, the processor (220) may determine the importance of the call based on volume-related action triggers. For example, the processor (220) may determine the importance of the call when, by user operation, the call volume becomes lower than the volume for multimedia content, the volume for multimedia content becomes lower than the call volume, or the volume is lowered below a specific volume level (or designated value).

[0087] According to one embodiment, the processor (220) may generate information corresponding to the other party's utterance when the importance of the call is high, and may display the generated information so as to be superimposed on multimedia content. According to one embodiment, the processor (220) may generate at least one of information corresponding to the other party's utterance, such as text or a graphic object, and may display at least one of the generated text or graphic object so as to be superimposed on at least a part of multimedia content.

[0088] According to one embodiment, the processor (220) can transmit an audio signal for the multimedia content and an audio signal corresponding to the other party's speech to an external electronic device (202) while displaying information generated on the multimedia content. In a multitasking state where a multimedia content application and a call application are executed simultaneously, the processor (220) can provide an audio signal corresponding to the multimedia content or the call to the user through the external electronic device (202) according to each situation. Here, the processor (220) may adjust only the volume of the audio signal corresponding to the call or only the volume of the audio signal corresponding to the multimedia content according to the importance of the call. In addition, the processor (220) may perform call muting in addition to adjusting the volume to be larger or smaller according to the importance of the call. For example, the processor (220) can control the call muting so that the other party's voice is not output through the external electronic device (202) so that the user can focus on listening to audio from the multimedia content while the multimedia content is being played when the importance of the call is low.

[0089] According to one embodiment, the processor (220) can check the importance (or priority) of a call in real time while outputting (or playing) multimedia content through the display (260). When the importance of the call is low, for example, when the priority of the multimedia content is higher than the priority of the call application, the processor (220) may control the volume of the audio signal corresponding to the other party's speech output through the external electronic device (202) to be lowered, and control the volume of the audio signal related to the multimedia content output through the external electronic device (202) to be higher.

[0090] According to one embodiment, the timing for determining the importance of a call may vary. According to one embodiment, information related to the call may be used to determine the importance of the call in real time during a call, and information related to multimedia content may also be used. For example, not only the call content that changes in real time but also highlight scenes of multimedia content may be used to determine the importance of the call. According to one embodiment, when user input for volume control is received to determine the importance of the call in real time during a call, it is possible to determine which the user prioritizes more between the call and the multimedia content depending on whether the operation is to increase the volume of the call or to increase the volume of the multimedia content. For example, if the user input for volume control corresponds to an input to increase the volume of the multimedia content or to control the call volume to 0, it can be determined that the importance (or priority) of the call is lower than the importance (or priority) of the multimedia content.

[0091] According to one embodiment, the processor (220) checks the importance (or priority) of a call in real time while in a multitasking state, and even if it simultaneously outputs an audio signal related to the multimedia content and an audio signal corresponding to the call through an external electronic device (202), it can generate information related to the call content at the moment when the importance of the call increases based on the result of analyzing information related to the call in real time using an AI model (231), and provide the generated information to the user. According to one embodiment, when the importance of the call increases, the processor (220) can display the generated information related to the call content so as to be superimposed on at least a part of the multimedia content. For example, the processor (220) can generate at least one of information corresponding to the other party's speech, such as text or a graphic object, and display at least one of the generated text or graphic object so as to be superimposed on at least a part of the multimedia content.

[0092] According to one embodiment, the processor (220) can perform actions according to the importance of the call in various ways, such as pausing the display of multimedia content or reducing the screen size of multimedia content, in addition to providing information related to the content of the call when the importance of the call increases.

[0093] According to one embodiment as described above, even if the volume of the audio signal corresponding to the call is lower than the volume of the audio signal related to the multimedia content or the volume of the audio signal corresponding to the call is 0, information related to the call content can be provided to the user in various forms such as text and graphic objects, thereby preventing the user from missing the call content while watching multimedia content.

[0094] According to one embodiment, the processor (220) may display a user interface that includes items for controlling the volume of each of the multimedia content application and the call application while displaying information related to call content generated on the multimedia content. Depending on the user selection through the user interface, the processor (220) may control the volume of the audio signal corresponding to the other party's speech output through the external electronic device (202) to increase, and control the volume of the audio signal related to the multimedia content output through the external electronic device (202) to decrease.

[0095] According to one embodiment, the processor (220) can adjust the volume of audio signals associated with multimedia content and calls, respectively, in addition to user input that manipulates the volume through a user interface, according to the importance of the call. For example, the processor (220) may determine the importance of the call in real time and control the volume of the audio signal corresponding to the other party's speech output through the external electronic device (202) to increase, and control the volume of the audio signal associated with the multimedia content output through the external electronic device (202) to decrease.

[0096] According to one embodiment, the processor (220) may determine the importance of a call and, for example, based on at least one piece of information related to the importance of the call, generate a prompt to display the call content as visual information (or visual feature) (e.g., text and graphic objects) according to the determined result, and transmit the generated prompt to the AI ​​model (231). Additionally, the processor (220) may generate a prompt to adjust the volume associated with the call according to the determined result and transmit the generated prompt to the AI ​​model (231). In this way, at the moment when the importance of the call increases, the call content may be displayed overlapping with at least a part of the multimedia content. Furthermore, while the user is watching the multimedia content, at the moment when the importance of the call increases, the volume associated with the call is increased so that the user can check the call content in real time and participate (or intervene) in the call with the other party. According to one embodiment, when the importance of the call increases, the information displayed regarding the call content may change. For example, when the importance of the call increases, the processor (220) can perform various actions, such as providing information that the user watching the multimedia content needs to participate in the call with the other party and speak, or providing audio for the call, and the types of actions that can be performed may vary depending on the importance of the call.

[0097] According to one embodiment, the processor (220) can use an AI model (231) to determine the importance of a call in real time while running a multimedia content application and a call application simultaneously, and can visually display content related to the call or adjust the volume related to the call, thereby increasing the efficiency of multitasking.

[0098] According to one embodiment, the memory (230) (e.g., the memory (130) of FIG. 1a) may store at least one instruction (or command) that causes at least one operation of the electronic device (201). When executed by the processor (220), the at least one instruction may cause the electronic device (201) to perform the corresponding operation. The memory (230) may store information such as data processing content of the processor (220) and training data of an on-device AI model.

[0099] According to one embodiment, the electronic device (201) may include an artificial intelligence (AI) model (231). According to one embodiment, the AI ​​model (231) is included in the electronic device (201) and may be referred to as an on-device AI model. For example, the AI ​​model (231) may be included in a memory (230).

[0100] According to one embodiment, when a situation occurs in which an audio signal (or audio data) is output by the playback of multimedia content and an audio signal is output by a call service, the processor (220) can identify at least one piece of information related to the importance of the call service and, based on the priority of the call service, drive an on-device AI model (231) (e.g., a machine learning model, a generative artificial intelligence model) to generate at least one of a summarized text or a graphic object based on text about the call content based on prompt input about the call content. The graphic object may be referred to as an image, icon, character, avatar, or indicator related to the call content.

[0101] Cases where an audio signal (or audio data) is output due to the playback of multimedia content and a voice signal is output due to a call service occur simultaneously may include cases where a call connection with the other party's electronic device occurs while an audio signal is being output due to the output (or playback) of multimedia content, or cases where a request for the playback of multimedia content is detected while an audio signal is being output due to a call connection with the other party's electronic device.

[0102] According to one embodiment, the processor (220) can generate text based on call content and graphic objects reflecting call content using an AI model (231), and display the generated text and graphic objects on at least a part of multimedia content (or multimedia content playback screen) through a display (260). Additionally, the processor (220) can determine the placement of text and graphic objects related to call content on the multimedia content playback screen using an AI model (231) based on the personal information of the electronic device (201), call content, or multimedia content (or attributes of the multimedia content). Additionally, the processor (220) can determine not only the position of the graphic object but also the shape of the graphic object using an AI model (231) based on the personal information of the electronic device (201), call content, or multimedia content (or attributes of the multimedia content).

[0103] According to one embodiment, the processor (220) can display multimedia content including information related to the call content (e.g., text and / or graphic objects) through the display (260).

[0104] According to one embodiment, the AI ​​model (231) may include at least one AI model (e.g., a first AI model (231a), or a second AI model (231b)).

[0105] According to one embodiment, the first AI model (231a) may include a machine learning model and may be an on-device AI model or an external AI model (281). For example, the first AI model (231a) may include a large language model (LLM), and may include various models such as a large multimodal model (LMM) and a large action model (LAM) in addition to the LLM, and the types of machine learning models may not be limited thereto.

[0106] For example, LLM learning may include pre-training and / or fine-tuning. Pre-training is the process of enabling the LLM to acquire general linguistic knowledge using large amounts of text data, and may include, for example, self-supervised learning that predicts the next word using the previous sequence of words in a sequence of text. Fine-tuning is the process of training the LLM to be suitable for a specific domain (e.g., chatbot, translation, summarization, Q&A) or task; based on the pre-trained model, the LLM can be further supervised (or adaptive) using a dataset tailored to the domain purpose. The LLM can perform tasks using text input containing natural language called a prompt.

[0107] For example, fine-tuning can be omitted during LLM training. Users can control the prompts input to the LLM to improve the performance of desired tasks. In a manner similar to in-context learning or zero-shot / few-shot learning, task examples and / or guides for performing the task can be added to the prompts. Examples of publicly available LLMs include BERT (bidirectional encoder representations from transformer) and GPT (generative pre-trained transformer).

[0108] The term 'LLM' can refer to the language neural network model itself, but it can also refer to models for LLM-based applications (e.g., chatbots, translation, summarization, text classification, or sentence generation). For instance, an LLM-based chatbot or translator like ChatGPT can also be referred to as 'LLM'.

[0109] 'LLM' may include an inference engine using an LLM neural network model. For example, "inputting an input prompt into the LLM" may mean "inputting an input prompt into an inference engine based on the LLM." For example, "the output of the LLM for the input prompt" may refer to the output information of the last neural network layer of the LLM (or output information modified through additional processing) obtained when the input prompt is input into an inference engine based on the LLM.

[0110] According to one embodiment, the second AI model (231b) may include a generative AI model and may be an on-device AI model or an external AI model (281).

[0111] According to one embodiment, the AI ​​model (231) of FIG. 2 has been described by dividing it into a first AI model (231a) and a second AI model (231b), but it may be configured as a single AI model without being divided into a first AI model (231a) and a second AI model (231b), and is not limited thereto, it may include other AI models necessary for generating text and graphic objects corresponding to call content, or it may not include a generative AI model. For example, the generative AI model may be included in an external electronic device (e.g., the electronic device (102, 104) or server (108) of FIG. 1a). According to one embodiment, when a generative AI model is included in an external electronic device, the electronic device (201) transmits a command for generating text and graphic objects corresponding to call content to the external electronic device through a communication circuit (290) (e.g., the communication module (190) of FIG. 1a), and can obtain text and graphic objects related to call content, which are the result generated by the generative AI model, from the external electronic device through the communication circuit (290).

[0112] According to one embodiment, the machine learning model may include a generative AI model trained using a neural network structure to generate and output text or graphic objects (or images) related to the call content based on prompt input. For example, an on-device AI model (or AI engine), such as a machine learning model, other machine learning models, and / or a generative AI model, may be stored in memory (230) or on an external device (e.g., a server (108)).

[0113] Meanwhile, in order to implement the present invention and operate it in an electronic device (201), hardware capable of multiple parallel operations, such as a GPU (Graphics processing unit) or an NPU (Neural network processing unit), may be required. Since most AI models, not just generative AI models, need to process a large number of arithmetic operations simultaneously, hardware capable of parallel operations rather than a CPU (Central processing unit) may be used, and this may be included in the electronic device (201) as a separate component (or module, circuit). As illustrated in FIG. 2, the case where the AI ​​model (231) is included in the form of hardware or a software block is exemplified, but the AI ​​model (231) may be included in memory (230) or a processor (220). For example, at least a portion of the AI ​​model (231) may be stored in the memory (230) of the electronic device (201) in the form of a program.

[0114] The aforementioned NPU may refer to a structure designed as a chip for operations to execute AI models (e.g., convolution, ReLU, Linear, BatchNorm, LayerNorm, Upsample, Transpose), while the GPU may refer to a structure capable of performing a large number of parallel operations based on a large number of cores. Typically, an NPU may have a structure designed to target a specific task (e.g., segmentation or tracking).

[0115] According to one embodiment, when a call (320) and content playback (330) are executed simultaneously, at least one piece of information (310) may be used to determine the importance of the call (320) and the importance of the content playback (330). The at least one piece of information (310) used to determine the importance (or priority) may include at least one of attribute information (311), volume information (313), personalization information (315), or attribute information (317) of an electronic device (201).

[0116] For example, the electronic device (201) can determine the priority between call connection and multimedia content playback based on stored volume information (313), but can also determine the priority between call connection and multimedia content playback based on volume operation by a user input in real time. For example, the electronic device (201) can determine that the priority of multimedia content playback is higher than the priority of call connection if the volume of multimedia content is set higher by volume operation by a user. Conversely, it can determine that the priority of call connection is higher than the priority of multimedia content playback if the volume of call connection is set higher by volume operation by a user.

[0117] For example, call importance can be set for each contact, and in the case of a user registered in the contact favorites, it may correspond to a case where call importance is set high. Therefore, the electronic device (201) can determine that when making a call with a counterpart registered in the contact favorites, the priority of connecting the call is higher than the priority of playing multimedia content.

[0118] According to one embodiment, the processor (220) can generate a prompt for call analysis based on at least one of attribute information (311), volume information (313), personalization information (315) of the electronic device (201) or attribute information of the external electronic device (202) using a prompt generation circuit (or prompt generation module) (340). Additionally, the processor (220) can perform call analysis (360) using an AI model (231) based on the generated prompt.

[0119] Refer to FIGS. 3B and FIG. 3C to explain a prompt used to generate information related to a call. FIG. 3B is a diagram illustrating the structure of a prompt for generating text related to a call according to one embodiment, and FIG. 3C is a diagram illustrating the structure of a prompt for generating a graphic object related to a call according to one embodiment.

[0120] A processor (220) according to one embodiment may generate a prompt for call content as in 300b based on a prompt structure as shown in 300a of FIG. 3b. For example, the prompt may include various information (or keywords) (e.g., call ID, current time, and / or speaker name) necessary to generate information (e.g., text) related to the call, and it is of course possible to configure it using various information in addition to the information shown in FIG. 3b.

[0121] A processor (220) according to one embodiment can generate a prompt (e.g., a graphic object) for call content as in 300d based on a prompt structure as shown in 300c of FIG. 3c. For example, the prompt may include various information (or keywords) (e.g., character request ID, current time, and / or profile picture image) necessary to generate information related to the call (e.g., a graphic object), and can be configured using various information as well as the information shown in FIG. 3c.

[0122] A processor (220) according to one embodiment may input a prompt requesting the creation of information related to a call (e.g., text, graphic object) into an AI model (231).

[0123] According to one embodiment, the operation (360) of analyzing a call can be performed in an electronic device (201). For example, during a call operation, a bidirectional stream channel (SCO) is opened between the electronic device (201) and the external electronic device (202) using a Bluetooth HFP profile, so the external electronic device (202) can transmit the other party's spoken sound source to the electronic device (201). The electronic device (201) can extract the other party's spoken sound source transmitted from the external electronic device (202) and can analyze the call content based on the extracted spoken sound source. The call content analyzed by the electronic device (201) can be used to determine the importance of the call. For example, the electronic device (201) can classify the analyzed call content into categories and determine the importance of the call using the importance of each category. For example, the electronic device (201) can determine that the call has high importance if the content requires the user to answer. As such, the operation (360) of analyzing a call has been described as an example of being performed in an electronic device (201), but it may also be performed in an external electronic device (202).

[0124] The AI ​​model (231) can generate and output information related to the call, such as text and / or graphic objects, based on the generated prompt input. According to one embodiment, the AI ​​model (231) can generate information (370) summarizing the call content based on the prompt input, and the processor (220) can display the call summary information (390) on at least a portion of the content playback screen (380). Although a separate prompt generation circuit (340) is shown in FIG. 3a, the operation of generating the prompt may be performed by the processor (220), and the operation of analyzing the call (360) may also be performed by the AI ​​model (231).

[0125] For example, the AI ​​model (231) can perform the task of generating descriptive text about the call content based on at least one of the attribute information (311), volume information (313), personalization information (315) of the electronic device (201) or the attribute information (317) of the external electronic device (202). For example, even if audio signals generated by content playback and audio signals related to a call are simultaneously transmitted to an external electronic device (202), if volume information (313) is checked and the volume during content playback is set higher than the volume during a call, the processor (220) can control the audio signals generated by content playback to be output through the speaker of the external electronic device (202) using personal information (e.g., usage pattern, or relationship with the other party), call-related information (e.g., name of the other party, photo of the other party, call content, or spam call), information of the content being played, or context information of the electronic device (201) (e.g., connection information with the external electronic device (202)), while controlling the audio signals related to the call not to be output through the speaker of the external electronic device (202) or to be output at a minimum volume.

[0126] For example, the processor (220) can be used to verify the identity of the caller to determine that advertising or spam calls have low importance, while calls from close contacts stored in the contacts have high importance. Additionally, the processor (220) can be used to check the call time to determine that work-related calls received during business hours have high importance, while non-work calls received during personal time have low importance. Additionally, the processor (220) can be used to look up the individual's recent conversation history to determine that calls with similar topics have high importance.

[0127] Additionally, the processor (220) monitors the call content and, if the call content requires user intervention (or a response), generates a prompt to display the call content as visual information, and can use the generated prompt as input to generate and output text and / or graphic objects for the call content from the AI ​​model (231).

[0128] According to one embodiment, the processor (220) can control text information and / or graphic objects regarding call content generated through the AI ​​model (231) to be displayed on a content playback screen or displayed through an external electronic device (202) according to user settings.

[0129] According to one embodiment, the electronic device (101, 201) may include a communication circuit (190, 290) including a first communication circuit and a second communication circuit, a display (160, 260), at least one processor (120, 220) including a processing circuit, and a memory (130, 230) for storing instructions.

[0130] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to connect with a first external electronic device (202) through the first communication circuit. For example, the first external electronic device may include a wearable electronic device.

[0131] According to one embodiment, the instructions may be configured to transmit an audio signal associated with the multimedia content to the first external electronic device through the first communication circuit while outputting the multimedia content through the display, when executed individually or collectively by the at least one processor.

[0132] According to one embodiment, the instructions may be configured to confirm a call connection with a second external electronic device through the second communication circuit when executed individually or collectively by the at least one processor.

[0133] According to one embodiment, the instructions may be configured to generate information corresponding to a user’s utterance of the second external electronic device in response to confirming the call connection while outputting the multimedia content through the display, when executed individually or collectively by the at least one processor.

[0134] According to one embodiment, the instructions may be configured to display information corresponding to the user's utterance through the display so as to be superimposed on at least a part of the multimedia content when executed individually or collectively by the at least one processor.

[0135] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device transmits an audio signal related to the multimedia content and / or an audio signal corresponding to the user's utterance to the first external electronic device through the first communication circuit in response to confirming the call connection, and when the importance of the call is low, the volume of the audio signal corresponding to the user's utterance output through the first external electronic device may be controlled to be lower than the volume of the audio signal related to the multimedia content. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device confirms the importance of the call in response to an input for volume control, and when the importance of the call is low, the audio signal corresponding to the user's utterance may not be output through the first external electronic device while information corresponding to the user's utterance is displayed through the display so as to be superimposed on at least a part of the multimedia content. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may display information about the call content (e.g., text and / or graphic objects) while transmitting an audio signal corresponding to the user's utterance and an audio signal related to the multimedia content to the first external electronic device.

[0136] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to generate summarized text based on information corresponding to the user's utterance using a generative AI (artificial intelligence) model. According to one embodiment, the information corresponding to the user's utterance that overlaps with at least a portion of the multimedia content may include the summarized text.

[0137] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to generate at least one of the summarized text or graphic object based on a prompt input regarding the call content according to the call connection using the generative AI model. According to one embodiment, the shape and location of the graphic object may be determined based on at least one of the personalization information of the electronic device, the call content, or the multimedia content.

[0138] According to one embodiment, the instructions may be configured such that, when executed individually or collectively by the at least one processor, the electronic device checks the importance of the call based on at least one piece of information related to the call while outputting the multimedia content through the display, and based on the importance of the call, information corresponding to the user's utterance is superimposed on at least a part of the multimedia content through the display.

[0139] The above instructions may be configured such that, when executed individually or collectively by the at least one processor, the electronic device uses the generative AI model to determine whether to generate information corresponding to the user's utterance based on at least one of information related to the multimedia content or at least one of information related to the call.

[0140] According to one embodiment, at least one piece of information related to the call may include at least one of the user's name, the user's phone number, the relationship with the user, the time of the call connection with the user, the volume of the call connection, or the content of the call according to the call connection.

[0141] According to one embodiment, information related to the multimedia content may include at least one of the type of the multimedia content, the user's recent viewing history of the electronic device related to the multimedia content, or a highlight scene of the multimedia content.

[0142] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to transmit data related to information corresponding to the user's utterance to the first external electronic device through the first communication circuit so that information corresponding to the user's utterance is displayed through the first external electronic device.

[0143] FIG. 4 is a flowchart of the operation of an electronic device for performing multitasking using a wearable electronic device according to one embodiment. Referring to FIG. 4, the operation method may include operations 405 through 425. Each operation of the operation method of FIG. 4 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1a and the electronic device (201) of FIG. 2)) and at least one processor of the electronic device (e.g., the processor (120) of FIG. 1a and the processor (220) of FIG. 2). In one embodiment, at least one of operations 405 through 425 may be omitted, the order of some operations may be changed, or other operations may be added. To aid in understanding the description of FIG. 4, the description will be explained with reference to FIG. 5 and FIG. 6. FIG. 5 is an example screen diagram of call content displayed on multimedia content according to one embodiment, and FIG. 6 is an example screen diagram of a summary of call content displayed on multimedia content according to one embodiment.

[0144] Referring to FIG. 4, in operation 405, the electronic device (201) can be connected to a first external electronic device (202). For example, the electronic device (201) can establish communication with the first external electronic device (202) (e.g., a wearable electronic device) through a communication circuit (290) (e.g., a first communication circuit) based on a first network (e.g., the first network (198) of FIG. 1a). According to one embodiment, the first external electronic device may include a device such as a speaker that can be connected to the electronic device (201), as well as a wearable electronic device such as an earphone.

[0145] In operation 410, the electronic device (201) can transmit an audio signal associated with the multimedia content to the first external electronic device while outputting the multimedia content. For example, the electronic device (201) can transmit an audio signal associated with the multimedia content to the wearable electronic device so that a user wearing the wearable electronic device can hear the audio of the multimedia content output through the speaker of the wearable electronic device while displaying the multimedia content through the display (260).

[0146] In operation 415, the electronic device (201) can confirm a call connection with a second external electronic device. According to one embodiment, the second external electronic device may be an electronic device of the other party communicating with the user of the electronic device (201).

[0147] According to one embodiment, the electronic device (201) can check whether there is a call connection request with a user of the second external electronic device (e.g., a counterparty). The call connection request may include receiving a call from the counterparty's second external electronic device or making a call from the electronic device (201). For example, in the case of receiving a call, the call connection with the second external electronic device can be checked by confirming whether the call is connected with the counterparty by selecting to accept the call, and in the case of making a call, the call connection with the second external electronic device can be checked by pressing the call button and confirming whether the call is connected with the counterparty.

[0148] According to one embodiment, the electronic device (201) may transmit an audio signal related to the multimedia content and / or an audio signal corresponding to the user's speech to the first external electronic device in response to confirming the call connection. The electronic device (201) may control the volume of the audio signal corresponding to the user's speech output through the first external electronic device to be lower than the volume of the audio signal related to the multimedia content when the importance of the call is low. For example, even if the electronic device (201) transmits the audio signal related to the multimedia content and / or the audio signal corresponding to the user's speech to the first external electronic device, it may control the output of each audio signal according to the importance of the call.

[0149] According to one embodiment, the electronic device (201) transmits an audio signal related to the multimedia content to the first external electronic device, but may not transmit an audio signal corresponding to the user's speech (e.g., the other party's speech) to the first external electronic device. For example, if the call volume is 0, the audio signal itself corresponding to the user's speech may not be transmitted to the first external electronic device. Additionally, even if the audio signal corresponding to the user's speech is transmitted to the first external electronic device along with the audio signal related to the multimedia content, only the audio signal related to the multimedia content may be output through the first external electronic device, and the audio signal corresponding to the user's speech may not be output through the first external electronic device.

[0150] According to one embodiment, the electronic device (201) may determine whether to transmit an audio signal corresponding to the user's speech to a first external electronic device during multitasking, or may determine whether to output an audio signal corresponding to the user's speech through the first external electronic device even if an audio signal corresponding to the user's speech is transmitted to the first external electronic device.

[0151] According to one embodiment, the electronic device (201) can determine the importance of a call and, depending on the importance of the call, determine whether to transmit an audio signal corresponding to the user's speech or to output an audio signal corresponding to the user's speech through a first external electronic device, but the criteria for determining whether to transmit and whether to output are not limited to those described above.

[0152] According to one embodiment, the electronic device (201) can determine the importance of a call and decide whether to generate information corresponding to the user’s utterance of the second external electronic device based on the importance of the call. For example, the electronic device (201) may automatically adjust the volume based on the importance of the call, but there may also be cases where the user directly adjusts the volume while performing multitasking.

[0153] According to one embodiment, the electronic device (201) can determine the importance of a call in response to an input for volume operation. For example, since the importance of a call may change due to the user's volume operation, the electronic device (201) can determine the importance of the call when the volume is operated and decide whether to generate information corresponding to the user's speech (e.g., the other party's speech) of the second external electronic device according to the importance of the call. Additionally, the electronic device (201) may configure the content included in the information corresponding to the user's speech of the second external electronic device differently according to the importance of the call. Furthermore, if the electronic device (201) determines that the importance of the call is low as a result of determining the importance of the call, it can control the volume of the audio signal corresponding to the user's speech output through the first external electronic device to be lower than the volume of the audio signal related to the multimedia content.

[0154] According to one embodiment, FIG. 4 describes a case where a call connection is checked during the playback of multimedia content, but it is obvious that the present invention can also be applied to an operation of determining whether output of multimedia content is requested during a call.

[0155] In operation 420, the electronic device (201) can generate information corresponding to the user’s speech of the second external electronic device in response to confirming the call connection while outputting the multimedia content.

[0156] In operation 425, the electronic device (201) may display information corresponding to the user's speech so as to be superimposed on at least a portion of the multimedia content. For example, referring to FIG. 5, the electronic device (201) may display text (520) corresponding to the speech of a user (e.g., a counterpart) on at least a portion of the multimedia content (510) displayed through the display (260). As illustrated in FIG. 5, the text (520) corresponding to the user's speech includes the content of the call and may be displayed superimposed on at least a portion of the multimedia content (510) so as not to obscure the multimedia content (510) as much as possible.

[0157] According to one embodiment, the electronic device (201) may generate summarized text based on information corresponding to the user's utterance using a generative AI (artificial intelligence) model. According to one embodiment, the information corresponding to the user's utterance superimposed on the multimedia content may include the summarized text. Additionally, the information corresponding to the user's utterance superimposed on the multimedia content may include text representing the entire content of the utterance. According to one embodiment, the text corresponding to the user's utterance may be configured differently depending on the information related to the call, for example, to represent the entire content of the utterance or in a summarized form. For example, referring to FIG. 6, the electronic device (201) may display summarized text (620) related to the call content superimposed on at least a portion of the multimedia content (610) displayed through the display (260).

[0158] According to one embodiment, the electronic device (201) may generate at least one of the summarized text or graphic object based on prompt input regarding the call content according to the call connection using the generative AI model. According to one embodiment, the shape and position of the graphic object may be determined based on at least one of the personalized information of the electronic device, the call content, or the multimedia content.

[0159] For example, when a call and multimedia content playback are performed simultaneously, the electronic device (201) can generate a prompt such as, "When the volume of the call changes to a level below a specified level, create an avatar using the caller information and provide summary information on at least part of the content playback screen."

[0160] According to one embodiment, the electronic device (201) may display a graphic object generated together with text corresponding to the user's utterance so as to be superimposed on the multimedia content. According to one embodiment, the electronic device (201) may display only the text corresponding to the user's utterance or only the graphic object, and the method of displaying the text and graphic object in relation to the user's utterance is not limited thereto.

[0161] According to one embodiment, the electronic device (201) can determine the importance of the call based on at least one piece of information related to the call in response to confirming the call connection while outputting the multimedia content. The electronic device (201) can display information corresponding to the user's utterance based on the importance of the call so as to be superimposed on at least a part of the multimedia content. For example, when determining the importance of the call based on at least one piece of information related to the call, if the information related to the call includes spam information, the call screen may not be displayed or the call connection itself may not be established. If the call connection is confirmed (or completed), an operation to determine the importance of the call based on at least one piece of information related to the call (e.g., call content) may be performed to determine whether to display information corresponding to the user's utterance (e.g., call content) in real time, even if the call connection is completed and the call is in progress.

[0162] According to one embodiment, the electronic device (201) can determine whether to generate information corresponding to the user's speech based on at least one piece of information related to the call or at least one piece of information related to the multimedia content using the generative AI model.

[0163] According to one embodiment, at least one piece of information related to the call may include at least one of the user's name, the user's phone number, the relationship with the user, the time of the call connection with the user, the volume of the call connection, or the content of the call connection. The electronic device (201) can use at least one piece of information related to the importance of the call to determine, for example, whether the user of the second external electronic device (e.g., the counterparty) is a user registered in the contact favorites of the electronic device (201), and can analyze the content of the call with the user to determine whether it is a call for making an important appointment. To determine the importance of the call, the electronic device (201) can determine, based on at least one piece of information related to the call, that the importance (or priority) of the call is higher than the importance (or priority) of the playback of the multimedia content, and accordingly, can use an AI model (231) to generate information corresponding to the user's speech and display it so as to be superimposed on at least a part of the multimedia content.

[0164] According to one embodiment, the audio signal of a call and the audio signal of multimedia content can be output simultaneously, and the volume can be adjusted separately. For example, when the importance of the call is high, the electronic device (201) can increase the volume of the audio signal of the call and lower or mute the volume of the audio signal of the multimedia content.

[0165] According to one embodiment, the electronic device (201) can check the importance of the call in response to an input for volume control. The electronic device (201) can control the output of an audio signal corresponding to the user's speech through the first external electronic device while displaying information corresponding to the user's speech overlaid on at least a part of the multimedia content when the importance of the call is low. For example, even if the electronic device (201) transmits an audio signal related to the multimedia content and an audio signal corresponding to the user's speech (e.g., the other party's speech) to the first external electronic device, it can control the output of only the audio signal related to the multimedia content through the first external electronic device and not output the audio signal corresponding to the user's speech (e.g., the other party's speech) through the first external electronic device. Additionally, the electronic device (201) can control the output of only the audio signal related to the multimedia content to the first external electronic device so that the audio signal corresponding to the user's speech (e.g., the other party's speech) is not output through the first external electronic device. For example, so that the user can focus on listening to the audio from the multimedia content, the voice corresponding to the user's speech during a call (e.g., the other party's voice) can be controlled so that it is not output through an external electronic device (202).

[0166] According to one embodiment, while the electronic device (201) displays information corresponding to the user's speech so as to be superimposed on at least a portion of the multimedia content, if the priority for playing the multimedia content is higher than the call service, the volume of the audio signal corresponding to the user's speech output through the external electronic device (202) may be controlled to decrease, and the volume of the audio signal related to the multimedia content output through the external electronic device (202) may be controlled to increase. For example, if the priority for playing the multimedia content is higher than the call service, the multimedia content is displayed through the display (260), and while the audio signal of the multimedia content is output through the speaker of the external electronic device (202), the voice corresponding to the user's speech during a call may be controlled not to be output through the external electronic device (202) or may be controlled to be output at a minimum volume so that the user is not disturbed by the other party's voice while listening to the audio of the multimedia content.

[0167] For example, if the priority of the call service becomes higher than the priority for the playback of multimedia content, information corresponding to the user's speech may be displayed so as to be superimposed on at least a part of the multimedia content, and the volume of the audio signal corresponding to the user's speech output through the external electronic device (202) may be controlled to increase so that the user can hear the call content well, and the volume of the audio signal related to the multimedia content output through the external electronic device (202) may be controlled to decrease.

[0168] According to one embodiment, the electronic device (201) can display summarized information related to the call content on the multimedia content based on information corresponding to the user's utterance using an AI model (231). Additionally, the electronic device (201) can control the volume of the call service and the volume during playback of the multimedia content while displaying summarized information related to the call content on the multimedia content using an AI model (231). Additionally, the electronic device (201) can display summarized information related to the call content on the multimedia content using an AI model (231) without volume control.

[0169] According to one embodiment, the electronic device (201) may determine whether to generate information corresponding to the user's speech based on at least one of the information related to the multimedia content in addition to at least one of the information related to the call.

[0170] According to one embodiment, information related to the multimedia content may include at least one of the type of the multimedia content, the user's recent viewing history of the electronic device related to the multimedia content, or a highlight scene of the multimedia content. For example, the electronic device (201) may use the type of multimedia content among the information related to the multimedia content to determine the importance of the multimedia content. For example, if the multimedia content requires concentration, such as a class lecture, or is an important scene that must not be missed, such as a highlight scene in a movie, the importance of playing the multimedia content is high, and if the content is for advertising or entertainment purposes, the importance may be relatively low. In addition, information such as whether the same content is viewed again by checking the recent video viewing history, or whether there is a high probability of watching content of interest with anticipation, may also be used to determine the importance of playing the multimedia content, and the types of information related to the multimedia content may not be limited thereto.

[0171] According to one embodiment, the electronic device (201) can determine the importance of a call by quantifying each of the information related to multimedia content and at least one piece of information related to a call to confirm (or determine) the importance of the call, and using the quantified result. Additionally, the electronic device (201) can generate a prompt to produce the quantified result and can determine which operation is more important between connecting the call and playing multimedia content based on the quantified result.

[0172] According to one embodiment, the electronic device (201) can control the audio signal corresponding to the user's speech of the second external electronic device and the audio signal of the multimedia content to be simultaneously transmitted to the external electronic device (202) based on the fact that the priority of the call is higher than the priority of the multimedia content playback, so that the audio signal corresponding to the multimedia content playback is not output through the external electronic device (202).

[0173] For example, the electronic device (201) can confirm that the user is a user registered in the contacts of the electronic device (201) based on at least one piece of information related to the call, and can confirm that the multimedia content being played through the display (260) includes a highlight scene based on information related to the multimedia content. For example, if the multimedia content includes a highlight scene, it can be determined that the priority of multimedia content playback is higher, and the electronic device (201) can control the volume of the audio signal corresponding to the user's speech output through the external electronic device (202) to be reduced or muted, and control the volume of the audio signal related to the multimedia content output through the external electronic device (202) to be increased.

[0174] According to one embodiment, if the electronic device (201) determines that the multimedia content includes a highlight scene and therefore has a higher priority for multimedia content playback, it can control the external electronic device (202) so that an audio signal corresponding to the user's speech is not output.

[0175] According to one embodiment, even if the electronic device (201) generates information corresponding to a user's utterance using an AI model (231), it may not display the generated information (e.g., text) corresponding to the user's utterance on the multimedia content when it is determined that the priority of multimedia content playback is higher because a highlight scene is included. For example, the electronic device (201) displays information summarizing the call content in real time using an AI model (231) when the priority of the call connection is higher than the priority of multimedia content playback, but when a highlight scene is included during multimedia content playback, the priority of multimedia content playback is higher, so even if information summarizing the call content is generated, the display of the summarized information may be restricted so as not to interfere with the viewing of multimedia content.

[0176] Those skilled in the art will understand that the embodiments described herein may be applied interchangeably to the extent applicable. For example, those skilled in the art will understand that at least some operations of an embodiment described herein may be omitted, and at least some operations of the embodiments may be applied interchangeably.

[0177] The present disclosure is not limited to the foregoing, and other unmentioned variations will be apparent to those skilled in the art from the present disclosure.

[0178] According to one embodiment, FIG. 4 describes an example in which, while an audio signal related to multimedia content is output through a first external electronic device, the importance of the call is checked and information related to the call content is displayed in at least a part of the multimedia content according to the importance of the call. However, the method of displaying information related to the call content in at least a part of the multimedia content according to the importance of the call can also be applied when the multimedia content is played while the call is connected first.

[0179] FIG. 7 is a first operation flowchart of an electronic device when simultaneously performing a call and multimedia content playback according to one embodiment. Referring to FIG. 7, the operation method may include operations 705 through 760. Each operation of the operation method of FIG. 7 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1a and the electronic device (201) of FIG. 2)) and at least one processor of the electronic device (e.g., the processor (120) of FIG. 1a and the processor (220) of FIG. 2). In one embodiment, at least one of operations 705 through 760 may be omitted, the order of some operations may be changed, or other operations may be added. Operations other than those of FIG. 7 may be performed before, during, or after the execution of the operations of FIG. 7. In FIG. 7, multimedia content may be referred to as content for convenience of explanation.

[0180] Referring to FIG. 7, in operation 705, the electronic device (201) can perform a call and content playback simultaneously. For example, the electronic device (201) can display a content playback screen through a display (260) and transmit an audio signal from the content playback and an audio signal from the call to be output through an external electronic device (202). Additionally, when a call connection is detected, it can perform an operation to determine the importance (or priority) between the call and the content playback. For example, while the call and content playback are being performed simultaneously, the user of the electronic device (201) can hear the sound from the content playback and the call sound output through the external electronic device (202).

[0181] In operation 710, the electronic device (201) can monitor the call content. The electronic device (201) can analyze the call content by monitoring the call content.

[0182] In operation 715, the electronic device (201) can check whether the importance of the call is high.

[0183] According to one embodiment, the electronic device (201) processes the audio signal from content playback and the audio signal of a call to be output through an external electronic device (202), and can determine the importance of the call by analyzing the call content using an AI model (231) (or a generative AI model) with the audio signal of the call as input in real time. According to one embodiment, the importance of the call can be determined by analyzing the call content using an AI model (231) (or a generative AI model) with the audio signal of the call and call-related information as input. Additionally, the electronic device (201) may determine the importance of the call by using a separate call analysis module (or call analysis circuit).

[0184] If, as a result of checking the importance of the call, the call is of low importance, the electronic device (201) may briefly provide information about the status of the call in progress. For example, the electronic device (201) may display an avatar representing the call partner, but may display an avatar that is motionless or has minimal movement. According to one embodiment, if the call is of low importance, the display of information related to the call may be limited, so the content of the call may not be displayed.

[0185] According to one embodiment, the electronic device (201) can determine whether to generate information (e.g., text and / or graphic objects) based on at least one piece of information related to a call or at least some of the information related to content using an AI model (231).

[0186] According to one embodiment, FIG. 7 describes an example in which the importance (or priority) of a call is checked by monitoring the call content, but the priority between the call content and the content being played can be checked simultaneously by monitoring both the call content and the content being played. For example, the electronic device (201) can analyze the content of the call and the content being played and check in real time whether there is a need to generate information related to the call content. For example, the electronic device (201) can check the importance (or priority) of the call and the importance (or priority) of the content while monitoring the call content.

[0187] In operation 715, the electronic device (201), based on confirming that the importance of the call is high, generates a first prompt to display visual information corresponding to the utterance of the call partner in operation 720, and in operation 725, transmits the first prompt to a generative AI model (e.g., the AI ​​model (231) of FIG. 2). FIG. 7 illustrates the case where one prompt is generated, but the electronic device (201) may generate multiple prompts.

[0188] In operation 730, the electronic device (201) can generate information about the content of a call (e.g., text and / or graphic objects) using a generative AI model. In operation 735, the electronic device (201) can display information about the content of a call (e.g., text and / or graphic objects) on multimedia content.

[0189] Meanwhile, in operation 715, if the electronic device (201) determines that the importance of the call is not high while monitoring the call content in real time, it may generate a second prompt for volume control in operation 750. Additionally, the electronic device (201) may generate a second prompt for volume control in operation 750 while displaying information about the call content (e.g., text and / or graphic objects) on the content in operation 735. In operation 755, the electronic device (201) may pass the second prompt to a generative AI model (e.g., the AI ​​model (231) of FIG. 2) to perform volume control in operation 760. For example, the electronic device (201) displays a content playback screen through the display (260), and when it is determined that the importance of the call is low while the audio signal from the content playback and the audio signal from the call are simultaneously output through the external electronic device (202), the volume of the audio signal from the call can be adjusted to be lower than the volume of the audio signal from the content playback or muted. For example, while the electronic device (201) displays a content playback screen through the display (260), when it is determined that the importance of the call is high while the audio signal from the content playback and the audio signal from the call are simultaneously output through the external electronic device (202), the volume of the audio signal from the call can be set higher than the volume of the audio signal from the content playback or the volume of the audio signal from the content playback can be adjusted to be muted.

[0190] According to one embodiment, FIG. 7 describes an example in which a volume adjustment operation is performed in operation 750 to 760 when the importance of the call is low; however, when the importance of the call is low, the display of information related to the call content may be minimized or the information related to the call content itself may not be displayed. For example, when the importance of the call is low, the electronic device (201) may display only limited content rather than the entire content of the other party's speech. For example, when the importance of the call is low, the electronic device (201) may be in a call with a counterparty related to the user of the electronic device (201), such as a credit card company or an insurance company, when the importance of the call is low, the counterparty may be speaking promotional content. Therefore, when the importance of the call is low, the volume of the counterparty's voice may be adjusted low, but the information related to the call may be composed of content different from the counterparty's speech, so that information indicating that the counterparty is promoting may be displayed or the counterparty's speech content itself may not be displayed. On the other hand, if the importance of the call is low, the volume of the other party's voice is lowered, or only limited information is displayed or not displayed at all, but the importance of the call increases as a result of monitoring the call content, this may be because the other party made an utterance requiring a response from the user, such as a request for identity verification, while speaking promotional content. Therefore, when the importance of the call increases, information related to the content of the other party's utterance, such as information requiring a response from the user, can be generated and displayed.

[0191] According to one embodiment, FIG. 7 describes an example in which the operation of monitoring call content is performed while displaying the content playback screen, but the electronic device (201) may monitor both the call content and the content being played and simultaneously check the priority between the call content and the content being played.

[0192] FIG. 8 is a second operation flowchart of an electronic device when simultaneously performing a call and multimedia content playback according to one embodiment. Referring to FIG. 8, the operation method may include operations 805 through 870. Each operation of the operation method of FIG. 8 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1a and the electronic device (201) of FIG. 2)) and at least one processor of the electronic device (e.g., the processor (120) of FIG. 1a and the processor (220) of FIG. 2). In one embodiment, at least one of operations 805 through 870 may be omitted, the order of some operations may be changed, or other operations may be added. Operations other than those of FIG. 8 may be performed before, during, or after the execution of the operations of FIG. 8. Additionally, operations 805 to 815 of FIG. 8 correspond to operations 705 to 715 of FIG. 7, operations 835 to 850 may correspond to operations 720 to 735 of FIG. 7, and operations 860 to 870 may correspond to operations 750 to 760 of FIG. 7. To aid in understanding the explanation of FIG. 8, the explanation will be described with reference to FIG. 9 to 14.

[0193] According to one embodiment, Figure 8 illustrates a case where volume control operations are performed in operations 860 to 870 when the importance of the call is low, but when the importance of the call is low, the display of information related to the call content may be minimized or the information related to the call content itself may not be displayed.

[0194] Referring to FIG. 8, in operation 805, the electronic device (201) can perform a call and content playback simultaneously. In operation 810, the electronic device (201) can monitor the call content. The electronic device (201) can analyze the call content by monitoring the call content. In operation 815, the electronic device (201) can determine whether the call is of high importance.

[0195] In operation 815, the electronic device (201) can determine whether a highlight scene is included during content playback in operation 825 based on confirming that the importance of the call is high. For example, the electronic device (201) can determine whether a highlight scene is included based on at least one piece of information related to the content being played while generating information related to the call content, as it is determined that the importance of the call is high. If a highlight scene is included, a delay can be added in operation 830. In operation 835, the electronic device (201) can generate a first prompt for displaying visual information and, in operation 840, transmit the first prompt to a generative AI model (e.g., the AI ​​model (231) of FIG. 2). In operation 845, the electronic device (201) can generate information about the call content (e.g., text and / or graphic objects) using the generative AI model and, in operation 850, display information about the call content (e.g., text and / or graphic objects) on the multimedia content. For example, the electronic device (201) can control that information about the content of a call (e.g., text and / or graphic objects) be displayed on at least part of the highlight scene at regular intervals, because the importance of the content is higher than the importance of the call in the case of a highlight scene. For example, the electronic device (201) can display information about the content of a call after the highlight scene has passed (or when the importance of the content decreases) while displaying the content.

[0196] Meanwhile, in operation 815, if the electronic device (201) determines that the importance of the call is not high while monitoring the call content in real time, it may generate a second prompt for volume control in operation 865. Additionally, the electronic device (201) may generate a second prompt for volume control in operation 860 while displaying information about the call content (e.g., text and / or graphic objects) on the content in operation 850. In operation 865, the electronic device (201) may pass the second prompt to a generative AI model (e.g., the AI ​​model (231) of FIG. 2) to perform volume control in operation 870. For example, the electronic device (201) displays a content playback screen through the display (260), and when it is determined that the importance of the call is low while the audio signal from the content playback and the audio signal from the call are simultaneously output through the external electronic device (202), the volume of the audio signal from the call can be adjusted to be lower than the volume of the audio signal from the content playback or muted. For example, while the electronic device (201) displays a content playback screen through the display (260), when it is determined that the importance of the call is high while the audio signal from the content playback and the audio signal from the call are simultaneously output through the external electronic device (202), the volume of the audio signal from the call can be set higher than the volume of the audio signal from the content playback or the volume of the audio signal from the content playback can be adjusted to be muted.

[0197] FIG. 9 is an example screen diagram showing call content displayed on multimedia content according to one embodiment using a first display method.

[0198] Referring to FIG. 9, the electronic device (201) may display visual information (or visual features) generated using a generative AI model (231) on at least a portion of the content playback screen (910). Text (920) displayed on at least a portion of the content playback screen (910) as in FIG. 9 may include the content of the other party's speech. When displaying the other party's speech, the electronic device (201) may display the entire speech content, but may also display text (920) that summarizes the long speech content as shown in FIG. 9. The summarized text (920) may be displayed by applying at least one of text size, boldness, color, and / or animation effects to text corresponding to important words so that the user can easily grasp the content related to the call. For example, in the case of call content corresponding to a question that requires the user's answer or intervention, the text size may be increased, the text color may be made bold, or effects such as shaking or tilting the text may be used so that the user can easily grasp the important content.

[0199] According to one embodiment, the electronic device (201) can use various graphic objects to inform the user of not only content related to the call but also situations where the user does not need to focus on the call. For example, various display or output methods can be used, such as muting the call when the content of the call is meaningless, changing the color of the screen to indicate when the user's response or intervention is required, or using a moving avatar to induce the user to listen to the content of the call.

[0200] FIG. 10 is an example screen diagram showing call content displayed on multimedia content using a second display method according to one embodiment, and FIG. 11 is an example screen diagram showing call content displayed on multimedia content using a third display method according to one embodiment.

[0201] Referring to FIGS. 10 and 11, the electronic device (201) may display a graphic object (1025, 1125) along with text (1020, 1120) related to the content of a call on a content playback screen (1010, 1110). The graphic object (1025, 1125) may include a character preferred by the user created using taste information among personalization information, or an avatar created using profile information of a contact's call partner. For example, in the case of a graphic object (1025) such as an avatar, the avatar's facial expressions and gestures may be used to make an expression that matches the content of the speech, or various effects may be used to induce user intervention. Additionally, the electronic device (201) may select a character corresponding to the call partner according to the user's selection. According to one embodiment, the graphic object (1025, 1125) may be used to provide a proxy response to the content of the call or to provide a visual warning instead of user intervention.

[0202] FIG. 12 is an example screen diagram showing call content displayed on multimedia content according to one embodiment using a fourth display method.

[0203] Referring to FIG. 12, the electronic device (201) can display text (1220) related to a call on a content playback screen (1210), but can also control the audio signal generated according to content playback to be output through the speaker of the external electronic device (e.g., wearable electronic device) (202) using context information of the electronic device (201) (e.g., connection information with an external electronic device (e.g., wearable electronic device) (202), and control the text related to the call to be output through the display (1230) of the external electronic device (e.g., wearable electronic device) (202).

[0204] For example, the electronic device (201) can determine where to display the content on the content playback screen using information about the content being played, and can also determine which device (e.g., an external electronic device (e.g., a wearable electronic device) (202), or another device connected to the electronic device (201)) to display it on. For example, when the content playback screen with subtitles (1215) is being displayed, the electronic device (201) can control the text related to the call content to be positioned at a location that does not overlap with the subtitles (1215). Here, the display location of the text related to the call content may be movable according to user input (e.g., touch followed by drag input).

[0205] FIG. 13 is a first screen example diagram for explaining the output volume of multimedia content and a call, respectively, according to one embodiment, and FIG. 14 is a second screen example diagram for explaining the output volume of multimedia content and a call, respectively, according to one embodiment.

[0206] Referring to FIG. 13, if the electronic device (201) determines that the importance (or priority) of the call service is higher than that of the content playback while displaying the content playback screen (1310), it can set the call volume high and adjust the content volume low. Conversely, if it determines that the importance of the content playback is higher than that of the call service, it can set the content volume high and adjust the call volume low. For example, the electronic device (201) can display the volume status for the content and the call, respectively, using an object (1320) representing the content volume and an object (1330) representing the call volume. For example, the electronic device (201) can adjust the volume using a volume value optimized for each user.

[0207] Referring to FIG. 14, if the electronic device (201) determines that the importance (or priority) of content playback is higher than that of the call service while displaying the content playback screen (1410), it can use an AI model (231) to generate text corresponding to the other party's speech on the content playback screen (1410) and display the generated text (1420) on the content playback screen (1410). Additionally, the electronic device (201) can use an AI model (231) to generate a graphic object corresponding to the other party (e.g., an image, icon, character, avatar, or indicator related to the call content) and display the generated graphic object (1425) on the content playback screen (1410). Furthermore, the electronic device (201) can display volume status for content and call, respectively, using an object (1430) representing content volume and an object (1440) representing call volume.

[0208] According to one embodiment, in a situation where services and multimedia content playback must be performed simultaneously, AI can be used to determine the importance of a call in real time to adjust the volume or allow the user to visually check the content of the call.

[0209] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0210] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0211] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0212] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0213] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0214] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0215] According to one embodiment, in a non-transient storage medium storing computer-readable instructions, the instructions are configured to cause the electronic device to perform at least one operation when executed by at least one processor (120, 220) of the electronic device (101), wherein the at least one operation may include: connecting to a first external electronic device (202); transmitting an audio signal related to the multimedia content to the first external electronic device while outputting the multimedia content; confirming a call connection with a second external electronic device; generating information corresponding to the utterance of the user of the second external electronic device in response to confirming the call connection while outputting the multimedia content; and displaying the information corresponding to the utterance of the user so as to be superimposed on at least a part of the multimedia content.

Claims

1. In an electronic device (101, 201), A communication circuit (190, 290) including a first communication circuit and a second communication circuit; Display(160, 260); At least one processor (120, 220) including a processing circuit; and It includes memory (130, 230) for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Connected to the first external electronic device (202) through the first communication circuit above, While outputting multimedia content through the above display, an audio signal related to the multimedia content is transmitted to the first external electronic device through the first communication circuit, and Confirming the call connection with the second external electronic device through the second communication circuit, In response to confirming the call connection while outputting the multimedia content through the display, information corresponding to the user's utterance of the second external electronic device is generated, and An electronic device configured to display information corresponding to the utterance of the user through the display so as to be superimposed on at least a portion of the multimedia content.

2. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, In response to confirming the above call connection, an audio signal related to the multimedia content and / or an audio signal corresponding to the user's speech is transmitted to the first external electronic device through the first communication circuit, and An electronic device that, when the importance of the above call is low, controls the volume of the audio signal corresponding to the user's speech output through the first external electronic device to be lower than the volume of the audio signal related to the multimedia content.

3. In claim 1 or 2, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Check the importance of the above call in response to input for volume control, and An electronic device that controls not to output an audio signal corresponding to the user's utterance through the first external electronic device while displaying information corresponding to the user's utterance through the display so as to overlap at least a part of the multimedia content when the importance of the above call is low.

4. In any one of claims 1 to 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Summarized text is generated using a generative AI (artificial intelligence) model based on information corresponding to the above user's utterance, and An electronic device comprising information corresponding to the user's utterance that overlaps at least a portion of the multimedia content, the summarized text.

5. In any one of claims 1 to 4, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Using the above generative AI model, at least one of the above summarized text or graphic object is generated based on prompt input regarding the call content according to the above call connection, and An electronic device in which the shape and position of the graphic object are determined based on at least one of the personal information of the electronic device, the call content, or the multimedia content.

6. In any one of claims 1 to 5, when the instructions are executed individually or collectively by the at least one processor, the electronic device, In response to checking the call connection while outputting the multimedia content through the display, the importance of the call is determined based on at least one piece of information related to the call, and An electronic device configured to display information corresponding to the user's utterance through the display so as to be superimposed on at least a portion of the multimedia content, based on the importance of the above call.

7. In any one of claims 1 to 6, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device configured to determine whether to generate information corresponding to the user's utterance based on at least one of information related to the multimedia content or at least one of information related to the call, using the generative AI model.

8. In any one of paragraphs 1 through 7, at least one piece of information related to the call is, An electronic device comprising at least one of the name of the user, the phone number of the user, the relationship with the user, the time of the call connection with the user, the volume of the call connection, or the content of the call according to the call connection.

9. In any one of paragraphs 1 through 8, the information related to the multimedia content is An electronic device comprising at least one of the type of multimedia content, the user's recent viewing history of the electronic device related to the multimedia content, or a highlight scene of the multimedia content.

10. In any one of claims 1 to 9, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device configured to transmit data related to information corresponding to the user's utterance to the first external electronic device through the first communication circuit so that information corresponding to the user's utterance is displayed through the first external electronic device.

11. A method for performing multitasking in an electronic device (101, 201), Operation of connecting to the first external electronic device (202); The operation of transmitting an audio signal associated with the multimedia content to the first external electronic device while outputting the multimedia content; An operation to confirm a call connection with a second external electronic device; An operation of generating information corresponding to the user’s utterance of the second external electronic device in response to confirming the call connection while outputting the multimedia content; and A method for performing multitasking, comprising the operation of displaying information corresponding to the user’s utterance so as to be superimposed on at least a portion of the multimedia content.

12. In Paragraph 11, In response to confirming the call connection, the operation of transmitting an audio signal related to the multimedia content and / or an audio signal corresponding to the user's speech to the first external electronic device; and A method for performing multitasking, further comprising, when the importance of the above call is low, controlling the volume of the audio signal corresponding to the user’s speech output through the first external electronic device to be lower than the volume of the audio signal related to the multimedia content.

13. In Paragraph 11 or 12, An operation to check the importance of the above call in response to an input for volume control; A method for performing multitasking, further comprising the operation of controlling not to output an audio signal corresponding to the user's utterance through the first external electronic device while displaying information corresponding to the user's utterance so as to be superimposed on at least a part of the multimedia content when the importance of the above call is low.

14. In any one of paragraphs 11 to 13, the operation of generating information corresponding to the utterance of a user of the second external electronic device is, It includes an operation to generate summarized text using a generative AI (artificial intelligence) model based on information corresponding to the utterance of the user, and A method for performing multitasking, wherein information corresponding to the user’s utterance, which overlaps with at least a portion of the multimedia content, includes the summarized text.

15. In a non-transient storage medium storing computer-readable instructions, said instructions are configured to cause said electronic device to perform at least one operation when executed by at least one processor (120, 220) of said electronic device (101, 201), said at least one operation being, said at least one operation Operation of connecting to the first external electronic device (202); The operation of transmitting an audio signal associated with the multimedia content to the first external electronic device while outputting the multimedia content; An operation to confirm a call connection with a second external electronic device; An operation of generating information corresponding to the user’s utterance of the second external electronic device in response to confirming the call connection while outputting the multimedia content; and A storage medium comprising an operation of displaying information corresponding to the utterance of the user so as to be superimposed on at least a portion of the multimedia content.

Images