Method for performing video call, and electronic device supporting same

Abstract

An electronic device disclosed in the present document may comprise a camera module, a display, a communication module, a memory and at least one processor. The electronic device can: identify whether a screen sharing function is executed during a video call; on the basis of a real-time transport protocol (RTP), transmit, to an external device, information related to a second resolution that is related to screen sharing and differs from a first resolution for a video call, when the screen sharing function is executed; generate second image frames by capturing an image being displayed on the display at the second resolution; and transmit the second image frames to the external device.

Description

Method for performing a video call and an electronic device supporting it

[0001] The embodiments disclosed in this document relate to a method for performing a video call and an electronic device supporting the same.

[0002] An electronic device (or portable communication device), such as a smartphone or tablet PC, can perform video calls (e.g., ViLTE, 5G video calls). For example, a first electronic device can acquire image frames captured through a camera. The first electronic device can convert the acquired image frames to a resolution pre-negotiated for video calls, and can compress and encode the converted image frames. The first electronic device can convert the encoded data into packets according to a specified protocol (e.g., RTP (real-time transport protocol)) and transmit them to a second electronic device. The second electronic device can also generate packets for video calls in the same manner as the first electronic device and transmit them to the first electronic device.

[0003] Various additional functions may be executed while a video call is in progress. For example, while a video call is in progress, the first electronic device may share the screen displayed on the display with the second electronic device. The first electronic device may capture the screen displayed on the display and transmit the captured image to the second electronic device for sharing. When the screen sharing function is executed during a video call, the data source may switch from image frames acquired through the camera to image frames captured from the screen displayed on the display.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] An electronic device according to one embodiment may include at least one processor comprising a camera module, a display, a communication module, a memory, and processing circuitry. The memory may store instructions that cause the electronic device to perform the following operations when executed individually or collectively by the at least one processor. The electronic device may acquire first image frames of a first resolution using the camera module. The electronic device may establish a video call by transmitting the first image frames to an external device using the communication module. During the video call, the electronic device may check whether a screen sharing function is executed. If the screen sharing function is executed, the electronic device may transmit information regarding a second resolution, which is related to screen sharing and differs from the first resolution, to the external device based on a real-time transport protocol (RTP). The electronic device may generate second image frames by capturing an image displayed on the display at the second resolution. The electronic device may transmit the second image frames to the external device.

[0006] A method for performing a video call according to one embodiment may be performed in an electronic device. A method for performing wireless communication may include: acquiring first image frames of a first resolution using a camera module of the electronic device; transmitting the first image frames to an external device using a communication module of the electronic device to establish a video call; checking whether a screen sharing function is executed while the video call is in progress; if the screen sharing function is executed, transmitting information regarding a second resolution different from the first resolution and related to screen sharing based on RTP (real-time transport protocol) to the external device; capturing an image displayed on the display of the electronic device at the second resolution to generate second image frames; and transmitting the second image frames to the external device.

[0007] A computer-readable storage medium according to one embodiment may store instructions executable by a processor. When the instructions are executed, the processor of the electronic device may perform the following operations. The processor may acquire first image frames of a first resolution using the camera module of the electronic device. The processor may establish a video call by transmitting the first image frames to an external device using the communication module of the electronic device. While the video call is in progress, the processor may check whether a screen sharing function is executed. If the screen sharing function is executed, the processor may transmit information regarding a second resolution different from the first resolution, which is related to screen sharing, to the external device based on the Real-Time Transport Protocol (RTP). The processor may generate second image frames by capturing an image displayed on the display of the electronic device at the second resolution. The processor may transmit the second image frames to the external device.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0009] FIG. 2a shows a video call between electronic devices according to one embodiment.

[0010] FIG. 2b shows screen sharing during a video call according to one embodiment.

[0011] FIG. 3 is a configuration diagram of a first electronic device and a second electronic device according to one embodiment.

[0012] FIG. 4a is a flowchart illustrating a method for performing a video call in a first electronic device according to one embodiment.

[0013] FIG. 4b is a flowchart illustrating a method for performing a video call in a second electronic device according to one embodiment.

[0014] FIG. 5 illustrates the transmission of second resolution information using an extension of the RTP header according to one embodiment.

[0015] FIG. 6 illustrates the transmission of second resolution information using an SEI message according to one embodiment.

[0016] FIG. 7a illustrates the transmission of second resolution information using RTP according to one embodiment.

[0017] FIG. 7b illustrates the transmission of second resolution information using RTP in a foldable device according to one embodiment.

[0018] FIG. 8a illustrates the transmission of second resolution information using RTCP according to one embodiment.

[0019] FIG. 8b illustrates the transmission of second resolution information using RTCP in a foldable device according to one embodiment.

[0020] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0021] Hereinafter, various embodiments of this document are described with reference to the accompanying drawings. However, this is not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives to the embodiments of this document. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0022] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (or display) (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0023] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0024] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0025] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0026] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0027] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0028] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0029] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0030] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0031] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0032] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0033] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0034] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0035] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0036] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0037] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0039] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0040] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0041] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0042] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0043] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0044]

[0045] FIG. 2a illustrates a video call between electronic devices according to one embodiment. FIG. 2a illustrates an exemplary case in which a video call is conducted using a front camera, but is not limited thereto.

[0046] Referring to FIG. 2a, the first electronic device (210) (e.g., the electronic device (101) of FIG. 1) can be connected to the second electronic device (220) (e.g., the electronic device (104) of FIG. 1) via wireless communication to perform a video call.

[0047] According to one embodiment, the first electronic device (210) may include a camera module (or camera, camera device) (211) (e.g., the camera module (180) of FIG. 1). The first electronic device (210) may acquire image frames captured through the camera module (211). The first electronic device (210) may convert the acquired image frames to a first resolution (e.g., HD (720x1280)) that is pre-negotiated for video calls, and may compress and encode the converted image frames. The first electronic device (210) may convert the encoded data into packets according to a specified protocol and transmit them to the second electronic device (220). For example, the first electronic device (210) may generate RTP (real-time transport protocol) packets and transmit them to the second electronic device (220) via a video channel (231). RTP (Real-time Transport Protocol) can be a protocol for transmitting audio and video data over an IP network in real time.

[0048] According to one embodiment, the first electronic device (210) can display an image transmitted from the second electronic device (220) over the entire area of ​​the display (212) and display an image captured through the camera module (211) of the first electronic device (210) over a part area of ​​the display (212).

[0049] According to one embodiment, the second electronic device (220) may include a camera module (221). The second electronic device (220) may acquire image frames captured through the camera module (221). The second electronic device (220) may convert the acquired image frames to a first resolution (e.g., HD (720x1280)) that is pre-negotiated for video calls, and may compress and encode the converted image frames. The second electronic device (220) may convert the encoded data into packets according to a specified protocol and transmit them to the first electronic device (210). For example, the second electronic device (220) may generate RTP (real-time transport protocol) packets and transmit them to the first electronic device (210) via a video channel (231).

[0050] According to one embodiment, the second electronic device (220) can display an image transmitted from the first electronic device (210) over the entire area of ​​the display (222) and display an image captured through the camera module (221) of the second electronic device (220) over a part area of ​​the display (222).

[0051] According to one embodiment, while the first electronic device (210) and the second electronic device (220) are conducting a video call, the first electronic device (210) may display a user interface (213) for executing various functions. The user interface (213) may include a button (214) that allows sharing a screen (or image frames) being displayed through a display (212) with the second electronic device (220). When a user touch input occurs on the button (214), the first electronic device (210) may transmit image frames captured from the display (212) of the first electronic device (210) to the second electronic device (220). Additional information regarding screen sharing during a video call may be provided through the drawings below.

[0052]

[0053] FIG. 2b shows screen sharing during a video call according to one embodiment.

[0054] Referring to FIG. 2b, when a screen sharing function is executed during a video call, the first electronic device (210) may transmit information regarding a second resolution related to screen sharing (hereinafter referred to as second resolution information) to the second electronic device (220) using the real-time transport protocol (RTP). The second resolution is the resolution of an image captured for screen sharing and may differ from the first resolution negotiated for the video call. For example, if the first resolution is 720x1280, the second resolution may be 600x1280.

[0055] According to one embodiment, the second resolution information may be information for extracting the second resolution. For example, the second resolution information may include a width value (e.g., 600) and a height value (e.g., 1280) of a shared image frame. Alternatively, the second resolution information may include an aspect ratio (e.g., 2.13) of the width x height of a shared image frame. When the first electronic device (210) transmits the second resolution information using a real-time transport protocol (RTP), the second electronic device (220) may extract the second resolution based on the second resolution information.

[0056] According to one embodiment, the second resolution may be set according to the size or aspect ratio of the display (212) of the first electronic device (210). For example, if the width (W1) x height (H1) of the display (212) of the first electronic device (210) has a resolution of 1080x2316, the second resolution may be set to 600x1280, which has the same or similar ratio as 1080x2316.

[0057] According to one embodiment, the first electronic device (210) can resize image frames captured at a second resolution (e.g., 600x1280) to convert them to a first resolution (e.g., 720x1280). The first electronic device (210) can transmit the resized image frames (e.g., 720x1280) to the second electronic device (220).

[0058] According to another embodiment, the first electronic device (210) can convert image frames captured at a second resolution (e.g., 592x1280) to a first resolution (e.g., 720x1280) by adding an offset (e.g., black band). The first electronic device (210) can transmit the image frames with the added offset to the second electronic device (220). The first electronic device (210) can transmit second resolution information including offset information to the second electronic device (220). In this case, the second electronic device (220) can extract the actual content area by removing the offset from the received image frames by referring to the offset information.

[0059] For example, if the second resolution is 592x1280 and the first resolution is 720x1280, the first electronic device (210) may add an offset (e.g., a black band) having a width of 64 pixels in each of the left / right directions (X-axis direction) of the captured image frame. In this case, the second resolution information containing the offset information may be offX (64), offY (0), outWidth (592), and outHeight (1280). The second electronic device (220) may extract the actual content area by removing the black band (an area having a width of 64 pixels at each of the left / right ends) from the received image frame by referring to the offset information (offX (64), offY (0)) in the second resolution information. The second electronic device (220) may extract the actual content area and transmit it to the application layer.

[0060] As another example, the second resolution information may include coordinate values ​​that are changed by the addition of an offset. The second resolution information may be four coordinate values: offX (64), offY (0), offX+outWidth (64+592), and offY+outHeight (0+1280). In this case, the second electronic device (220) can determine the content area of ​​the captured image frame (an area of ​​64 pixels to 656 pixels in the X-axis direction and an area of ​​0 pixels to 1280 pixels in the Y-axis direction) by determining the left / upper coordinate (offX, offY), right / upper coordinate (offX+outWidth, offY), left / lower coordinate (offX, offY+outHeight), and right / lower coordinate (offX+outWidth, offY+outHeight)), which are the coordinates of each corner of the content area in the image frame. The second electronic device (220) can extract the actual content area and transmit it to the application layer.

[0061] According to one embodiment, the second resolution may be determined differently depending on the state of the first electronic device (210). For example, if the first electronic device (210) is a foldable device, the second resolution may vary depending on whether it is in a folded or unfolded state. If the first electronic device (210) is a rollable device, the second resolution may vary depending on the unfolded area or aspect ratio of the display (212). Alternatively, if the first electronic device (210) includes a plurality of displays, the second resolution may vary depending on the size or aspect ratio of the display (212) that is displaying content.

[0062] According to one embodiment, when the first resolution (e.g., 720x1280) negotiated for video calls and the second resolution (e.g., 600x1280) for screen sharing are different, a captured image having the second resolution can be transmitted to the second electronic device (220) without transmitting the second resolution information. When the shared image is displayed on the display (222) of the second electronic device (220) having a width (W2) x height (H2) different from the display (212) of the first electronic device (210), distortion of the width x height ratio of the shared image or cropping of the corners of the shared image may occur due to the difference between the first resolution and the second resolution.

[0063] To prevent this, when screen sharing begins, the first electronic device (210) may transmit second resolution information to the second electronic device (220) based on RTP. The first electronic device (210) may transmit the second resolution information to the second electronic device (220) by including it in an RTP header or an SEI message. The second resolution information may be transmitted through a video channel (231) through which image frames of a video call are transmitted. Additional information regarding the transmission of the second resolution information may be provided through the drawings below.

[0064] According to one embodiment, the second electronic device (220) can render a shared video (e.g., image frames captured from the first electronic device (210)) based on second resolution information transmitted via RTP. For example, the second electronic device (220) can check whether the second resolution information is included in the RTP header or SEI message received from the first electronic device (210) during a video call at the first resolution (e.g., 720x1280). If the second resolution information is not included in the RTP header or SEI message, the second electronic device (220) can render the received RTP packets at the first resolution and process them as image frames for a normal video call. On the other hand, if the second resolution information is included in the RTP header or SEI message received from the first electronic device (210), the second electronic device (220) can obtain the second resolution using the second resolution information. The second electronic device (220) can render a shared image (e.g., image frames captured from the first electronic device (210)) at a second resolution. By doing so, the second electronic device (220) can display the shared image without aspect ratio distortion or cropping.

[0065] According to one embodiment, the captured image in the first electronic device (210) may include user interaction (e.g., drawing input (215)). For example, the drawing input (215) may be an input for selecting a specific object. When the shared image is rendered in the second electronic device (220) based on the second resolution information, the drawing input (215) may be displayed so that the same object is selected without distortion.

[0066] According to one embodiment, the second electronic device (220) can transmit information related to user interaction through a separate data channel (232) when a separate user interaction occurs. For example, when a drawing input (226) occurs while a shared image is being displayed, the second electronic device (220) can transmit coordinate information corresponding to a second resolution in relation to the drawing input (226) to the first electronic device (210). The first electronic device (210) can display the drawing input (226) displayed in the second electronic device (220) on the display (212) without distortion.

[0067]

[0068] FIG. 3 is a configuration diagram of a first electronic device and a second electronic device according to one embodiment. FIG. 3 is illustrated with a focus on configurations related to video calls, but is not limited thereto. In addition, the operation of each configuration of FIG. 3 may be an operation performed by a processor included in each device.

[0069] Referring to FIG. 3, the first electronic device (210) may include a video engine (310) for video calls. The video engine (310) may include a frame buffer (311), a size changer (312), an encoder (313), an SEI generation unit (314), and an RTP packet generation unit (315).

[0070] According to one embodiment, the frame buffer (311) may temporarily store image frames acquired through the camera module or image frames captured for screen sharing. The size change unit (or format change unit) (312) may convert image frames acquired through the camera module to a first resolution negotiated for video calls. The size change unit (312) may convert image frames captured for screen sharing to a first resolution. For example, the size change unit (312) may resize image frames captured at a second resolution (e.g., 600x1280) to convert them to a first resolution (e.g., 720x1280). Additionally, the size change unit (312) may perform color conversion (e.g., conversion from RGB to YUV) on the captured image frames.

[0071] The encoder (313) can encode image frames. The SEI generation unit (314) can generate an SEI message containing second resolution information. The RTP packet generation unit (315) can generate an RTP packet using the encoded image data. The RTP packet transmission unit (315) can generate packets containing second resolution information in the RTP header or SEI message.

[0072] According to one embodiment, the second electronic device (220) may include a video engine (320) for video calls. The video engine (320) may include an RTP packet receiver (321), an SEI parser (322), a decoder (323), a renderer (324), and a video call application (325).

[0073] According to one embodiment, an RTP packet receiver (321) can receive an RTP packet from a first electronic device (210). An SEI parser (322) can analyze the SEI message. A second electronic device (220) can check whether the SEI message contains second resolution information. If the SEI message contains second resolution information, the video call application (325) can be notified of the changed resolution (second resolution). A decoder (323) can decode the encoded image data. A renderer (324) can render the decoded image data to display the image.

[0074] According to one embodiment, since the video call application (325) does not know the resolution of the video rendered by the video engine (320), it can receive resolution information transmitted from the framework (e.g., onPeerDimension) and adjust the ratio of the surface view in response to the received resolution information. In the case of a normal video call rather than screen sharing, the framework can transmit information regarding a first resolution (e.g., 720x1280) determined through decoding to the video call application (325).

[0075] For example, if the resolution information received from the framework is 720x1280, the video call application (325) can adjust the surface view at a ratio of 9:16. As another example, if the resolution information received from the framework is 480x640, the video call application (325) can adjust the surface view at a ratio of 3:4.

[0076] According to one embodiment, when a screen sharing function is executed during a video call, the video call application (325) may receive second resolution information transmitted from a framework (e.g., onPeerDimension). The screen sharing application (325) may extract a screen ratio from the received second resolution information and adjust the size or ratio of a surface view displaying the shared video according to the extracted screen ratio.

[0077]

[0078] FIG. 4a is a flowchart illustrating a method for performing a video call in a first electronic device according to one embodiment. In FIG. 4a, the operation of the first electronic device (210) may be the operation of a processor included in the first electronic device (210) (e.g., the processor (120) of FIG. 1).

[0079] Referring to FIG. 2b and FIG. 4a, in operation 410, the first electronic device (210) can perform a video call with the second electronic device (220). The first electronic device (210) can transmit image frames (e.g., first image frames) acquired through the camera module (211) to the second electronic device (220) at a first resolution (e.g., HD (720x1280)) that has been pre-negotiated for the video call. The image frames can be transmitted as RTP packets. Additionally, the first electronic device (210) can render and display the RTP packets received from the second electronic device (220).

[0080] In operation 420, the first electronic device (210) can determine whether a screen sharing function is executed during a video call. For example, the first electronic device (210) may display a user interface (213) for executing various functions that can be executed during a video call. The user interface (213) may include a button (214) for sharing the screen being displayed through the display (212) with the second electronic device (220). When a user's touch input occurs on the button (214), the first electronic device (210) may determine that the screen sharing function is executed during a video call.

[0081] In operation 430, when the screen sharing function is executed (operation 420-YES), the first electronic device (210) may transmit information regarding a second resolution related to screen sharing (second resolution information) to the second electronic device (220) based on RTP. The first electronic device (210) may transmit the second resolution information to the second electronic device (220) by including it in an RTP header or an SEI message. The second resolution is the resolution of the captured image for screen sharing and may differ from the first resolution negotiated for the video call. For example, if the first resolution is 720x1280, the second resolution may be 600x1280.

[0082] According to one embodiment, the first electronic device (210) may continuously and repeatedly execute operation 430 while the screen sharing function is running. Alternatively, the first electronic device (210) may transmit second resolution information included in the first few RTP packets when a specified event occurs (e.g., execution of the screen sharing function or folding / unfolding of the foldable device). The first electronic device (210) may transmit first resolution information when the screen sharing function ends.

[0083] According to one embodiment, when the screen sharing function is not executed (operation 420-NO), the first electronic device (210) can continuously transmit image frames acquired through the camera module (211) to the second electronic device (220) at a first resolution (e.g., HD (720x1280)).

[0084] In operation 440, the first electronic device (210) can generate image frames (second image frames) captured at a second resolution and transmit them to the second electronic device (220). The image frames captured at the second resolution can be rendered at the second resolution in the second electronic device (220) (see FIG. 4b).

[0085] According to one embodiment, the first electronic device (210) is a foldable device, and when folding or unfolding occurs during the execution of a screen sharing function, the first electronic device (210) can change the value of the second resolution and transmit it to the second electronic device (220). For example, in the folding state, the resolution of the captured image (the second resolution) can be set to a first value (e.g., 600x1280). Subsequently, in the unfolding state, the resolution of the captured image (the second resolution) can be changed to a second value (e.g., 960x1280). The first electronic device (210) can transmit information regarding the second value of the second resolution (e.g., 960x1280) to the second electronic device (220) by including it in an RTP header or an SEI message.

[0086] According to one embodiment, the first electronic device (210) can check whether the screen sharing function is terminated. The first electronic device (210) can repeat operations 430 and 440 until the screen sharing function is terminated. When the screen sharing function is terminated, the first electronic device (210) may not include the second resolution information in the RTP header or SEI message. The first electronic device (210) can transmit image frames acquired through the camera module (211) to the second electronic device (220) at the first resolution (e.g., HD (720x1280)).

[0087]

[0088] FIG. 4b is a flowchart illustrating a method for performing a video call in a second electronic device according to one embodiment. In FIG. 4b, the operation of the second electronic device (220) may be the operation of a processor included in the second electronic device (220).

[0089] Referring to FIGS. 2b and FIGS. 4b, in operation 481, the second electronic device (220) can perform a video call with the first electronic device (210). The second electronic device (220) can transmit image frames acquired through the camera module (221) to the first electronic device (210) at a first resolution (e.g., HD (720x1280)) that has been pre-negotiated for the video call. The image frames can be transmitted as RTP packets. Additionally, the second electronic device (220) can render and display the RTP packets received from the first electronic device (210).

[0090] In operation 483, the second electronic device (220) can check whether second resolution information for screen sharing is included in the RTP header or SEI message. The second resolution information may include data indicating the resolution (second resolution) of image frames captured by the first electronic device (210).

[0091] In operation 485, if the second resolution information is included in the RTP header or SEI message (YES of operation 483), the second electronic device (220) can extract the second resolution from the second resolution information. The second resolution is the resolution of the captured image for screen sharing and may differ from the first resolution negotiated for the video call. For example, if the first resolution is 720x1280, the second resolution may be 600x1280.

[0092] According to one embodiment, if the second resolution information is not included in the RTP header or SEI message (NO of operation 483), the second electronic device (220) can continue to perform video calls with the first resolution.

[0093] In operation 487, the second electronic device (220) can render and display shared image frames to correspond to the second resolution. The second electronic device (220) can determine the size or ratio of the surface view to correspond to the second resolution. The second electronic device (220) can display image frames subsequently received through the surface view.

[0094] According to one embodiment, the second electronic device (220) may repeatedly perform operations 485 and 487 when it receives an RTP message or SEI message containing second resolution information from the first electronic device (210). When the screen sharing function is terminated, the second resolution information may not be included in the RTP header or SEI message. The second electronic device (220) may continue to perform video calls at the first resolution.

[0095]

[0096] FIG. 5 illustrates the transmission of second resolution information using an extension of the RTP header according to one embodiment.

[0097] Referring to FIG. 2b and FIG. 5, the first electronic device (210) can perform a video call with the second electronic device (220). During the video call, a screen sharing function can be executed by a separate user input.

[0098] According to one embodiment, when a screen sharing function is executed during a video call, the first electronic device (210) can transmit second resolution information (resolution information for screen sharing) to the second electronic device (220) using an extension of the RTP header (501). The second resolution may be the resolution of a captured image set in the first electronic device (210).

[0099] According to one embodiment, the extension of the RTP header (501) may be indicated as a URI in the SDP signal. For example, when the RTP header (501) is extended, the SDP signal may be indicated as "a=extmap:1 urn:ietf:params:rtp-hdrext:screenshare-video-resolution".

[0100] According to one embodiment, the RTP header (501) may include various information related to the RTP packet. The RTP header (501) may include a field (X) (505) regarding the header extension (510).

[0101] The header extension (510) may contain additional information regarding the RTP packet. The header extension (510) may be included if the extension indicator bit (X) (505) is 1, and may not be included if the extension indicator bit (X) (505) is 0. The header extension (510) may have a variable length.

[0102] The header extension (510) may include extension data (511) related to the second resolution. The extension data (511) may include an identifier (ID) (515), a length (len) (516), a first direction length (ss-video-width) (517), and a second direction length (ss-video-height) (518). For example, the first direction length (ss-video-width) (517) and the second direction length (ss-video-height) (518) each use 12 bits, and if the second resolution is 600 (W) x 1280 (H), the first direction length (ss-video-width) (517) may be set to 0010 0101 1000 and the second direction length (ss-video-height) (518) may be set to 0101 0000 0000.

[0103] According to one embodiment, the second electronic device (220) receives an RTP packet and can obtain a second resolution by referring to the second resolution information included in the header extension (510). The second electronic device (220) can extract the second resolution information included in the header extension (510) and transmit it to a video call application through a framework (e.g., onPeerDimension). Based on the second resolution, the video call application can display the shared video without aspect ratio distortion or cropping.

[0104]

[0105] FIG. 6 illustrates the transmission of second resolution information using an SEI message according to one embodiment.

[0106] Referring to FIG. 6, the first electronic device (210) can perform a video call with the second electronic device (220). During the video call, a screen sharing function can be executed by a separate user input.

[0107] According to one embodiment, the first electronic device (210) can transmit second resolution information (e.g., resolution information for screen sharing) to the second electronic device (220) using a supplemental enhancement information (SEI) message (601). The second resolution may be the resolution of a captured image set in the first electronic device (210).

[0108] According to one embodiment, the SEI message (601) is included in an RTP packet and may include additional information regarding the video or audio being transmitted. The SEI message (601) may be an additional message that is not an essential element required to decode the video. The first electronic device (210) may add second resolution information to the SEI message (601) with a specified data size (e.g., 10 Bytes).

[0109] According to one embodiment, the SEI message (601) may include NAL information (610), a payload type (620), a payload size (630), and a payload (640). The second resolution information may be included in the payload (640) of the SEI message (601). For example, the payload (640) may include 2-byte data representing the horizontal value of the second resolution and 2-byte data representing the vertical value of the second resolution.

[0110] According to one embodiment, when screen sharing is started, the first electronic device (210) may transmit an SEI message (601) before transmitting I frame among the image frames.

[0111] For example, before screen sharing starts, the first electronic device (210) can continuously transmit image data in the order of SPS (sequence parameter set), PPS (picture parameter set), I frame, and P frames consecutive to the I frame without a separate SEI message (601).

[0112] When screen sharing begins, the first electronic device (210) can continuously transmit image data in the order of SPS (sequence parameter set), PPS (picture parameter set), SEI message (601) (including second resolution information), I frame, and P frames consecutive to the I frame. The first electronic device (210) can continuously transmit the SEI message (601) including the second resolution information until screen sharing ends.

[0113] Afterward, when screen sharing ends, the first electronic device (210) can continuously transmit image data in the order of SPS (sequence parameter set), PPS (picture parameter set), I frame, and P frames consecutive to the I frame without a separate SEI message (601).

[0114] The second electronic device (220) can receive and parse the SEI message (601). The second electronic device (220) can transmit the second resolution information obtained through parsing to a video call application via a framework (e.g., onPeerDimension). Based on the second resolution, the video call application can display the shared video without aspect ratio distortion or cropping.

[0115]

[0116] FIG. 7a illustrates the transmission of second resolution information using RTP according to one embodiment.

[0117] Referring to FIG. 2b and FIG. 7a, the first electronic device (210) can perform a video call with the second electronic device (220). When the video call is initiated (710), the video source may be an image acquired through the camera module (211). The first electronic device (210) can transmit an image frame acquired in real time through the camera module (211) to the second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) can transmit an I frame and a P frame following the I frame. The I frame and the P frame are image data, and the I frame and the P frame following the I frame may form a group of pictures (GOP).

[0118] According to one embodiment, a screen sharing function may be executed during a video call (720). When the screen sharing function is executed, the video source may be changed from an image obtained through the camera module (211) to a captured image. For example, the resolution of the captured image (second resolution) (e.g., 600x1280) may be set differently depending on the size or aspect ratio of the display (211) of the first electronic device (210), or the state of the first electronic device (210) (e.g., folding / unfolding, or using the main display / using the sub display). The first electronic device (210) may include the second resolution information in an RTP header or SEI message and transmit it to the second electronic device (220).

[0119] With the start of the screen sharing function (720), the first electronic device (210) can transmit the captured image frames to the second electronic device (220). The captured image frames can be resized to a first resolution (e.g., HD (720x1280)) and transmitted to the second electronic device (220).

[0120] The first electronic device (210) can transmit an I frame and a P frame that follows the I frame. The I frame and the P frame are image data, and the I frame and the P frame can form a group of pictures (GOP). In the S1 interval where a GOP based on the second resolution is transmitted, the second electronic device (220) can process and display shared image frames at the second resolution by using the second resolution information (e.g., ratio value 2.13 of 600x1280) included in the RTP header or SEI message. Through this, aspect ratio distortion or corner cropping of the shared image due to a mismatch in resolution may not occur.

[0121] When screen sharing ends (730), the image source may be changed from a captured image to an image obtained through the camera module (211). The first electronic device (210) may transmit an image frame obtained in real time through the camera module (211) to the second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) may transmit an I frame and a P frame that follows the I frame.

[0122]

[0123] FIG. 7b illustrates the transmission of second resolution information using RTP in a foldable device according to one embodiment.

[0124] Referring to FIGS. 2b and FIGS. 7b, a first electronic device (210) can perform a video call with a second electronic device (220). The first electronic device (210) may be a foldable device. When a video call is initiated in a folded state (760), the video source may be an image acquired through a camera module (211). The first electronic device (210) may transmit an image frame acquired in real time through the camera module (211) to the second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) may transmit an I frame and a P frame that follows the I frame. The I frame and the P frame are image data, and the I frame and the P frame may form a group of pictures (GOP).

[0125] According to one embodiment, a screen sharing function may be executed by separate user input during a video call (770). When the screen sharing function is executed, the video source may be changed from an image obtained through the camera module (211) to a captured image. For example, the screen sharing function may be executed by user input when the first electronic device (210) is in a folded state. In the folded state, the resolution of the captured image (second resolution) may be set to a first value (e.g., 600x1280).

[0126] With the start of the screen sharing function (770), the first electronic device (210) may transmit to the second electronic device (220) information regarding a first value of the second resolution (e.g., 600x1280) (e.g., ratio value of 600x1280 2.13) in an RTP header or SEI message. The first electronic device (210) may transmit a captured image frame to the second electronic device (220). The first electronic device (210) may transmit an I frame and a P frame that follows the I frame. The second electronic device (220) may process and display the image frames transmitted thereafter at the first value of the second resolution (e.g., 600x1280).

[0127] According to one embodiment, when the first electronic device (210) changes from a folded state to an unfolded state (780), the resolution of the captured image (second resolution) may be changed to a second value (e.g., 720x960). The first electronic device (210) may include information regarding the second value of the second resolution (e.g., 720x960) (e.g., ratio value of 720x960 1.33) in an RTP header or SEI message and transmit it to the second electronic device (220). The first electronic device (210) may transmit the captured image frames to the second electronic device (220). The captured image frames may be resized to a first resolution (e.g., HD (720x1280)) and transmitted to the second electronic device (220).

[0128] The first electronic device (210) can transmit an I frame and a P frame that follows the I frame. The second electronic device (220) can process and display the subsequently transmitted image frames at a second value of a second resolution (e.g., 720x960).

[0129] In the S2 section where the GOP is transmitted at the second resolution (first value or second value), aspect ratio distortion or corner cropping of the shared image due to a resolution mismatch may not occur. Additionally, even when folding or unfolding occurs in the first electronic device (210) after the execution of the screen sharing function, aspect ratio distortion or corner cropping of the shared image due to a resolution mismatch may not occur.

[0130] When screen sharing ends (790), the image source may be changed from a captured image to an image obtained through the camera module (211). The first electronic device (210) may transmit an image frame obtained in real time through the camera module (211) to the second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) may transmit an I frame and a P frame that follows the I frame.

[0131]

[0132] FIG. 8a illustrates the transmission of second resolution information using RTCP according to one embodiment.

[0133] Referring to FIG. 2b and FIG. 8a, the first electronic device (210) can perform a video call with the second electronic device (220). When the video call is initiated (810), the video source may be an image acquired through the camera module (211). The first electronic device (210) can transmit an image frame acquired in real time through the camera module (211) to the second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) can transmit an I frame and a P frame that follows the I frame. The I frame and the P frame are image data, and the I frame and the P frame may form a group of pictures (GOP).

[0134] According to one embodiment, a screen sharing function may be executed during a video call (820). When the screen sharing function is executed (820), the video source may be changed from an image obtained through the camera module (211) to a captured image. The first electronic device (210) may transmit the second resolution information to the second electronic device (220) by including it in the source description (SDES) of the RTCP packet, rather than in the RTP header or SEI message.

[0135] With the start of the screen sharing function (820), the first electronic device (210) can transmit the captured image frames to the second electronic device (220). The captured image frames can be resized to a first resolution (e.g., HD (720x1280)) and transmitted to the second electronic device (220).

[0136] The first electronic device (210) can transmit an I frame and a P frame that follows the I frame. The I frame and the P frame are image data, and the I frame and the P frame can form a group of pictures (GOP).

[0137] If the second resolution information is included in the source description (SDES) of the RTCP packet, a period (D1) in which the user can perceive a change in resolution may occur due to the RTCP period (e.g., 2 sec). In the period (D1) after the start of the screen sharing function (820) and before receiving the RTCP (825), aspect ratio distortion or corner cropping of the shared image may occur due to a resolution mismatch. In the period (E1) after receiving the RTCP (825), aspect ratio distortion or corner cropping of the shared image due to a resolution mismatch may not occur.

[0138] When screen sharing ends (830), the image source may be changed from a captured image to an image obtained through the camera module (211). The first electronic device (210) may transmit an image frame obtained in real time through the camera module (211) to the second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) may transmit an I frame and a P frame that follows the I frame.

[0139] In the section (D2) after screen sharing ends (830) and before receiving RTCP (835), aspect ratio distortion or corner cropping of the image may occur due to a resolution mismatch. In the section (E2) after receiving RTCP (835), aspect ratio distortion or corner cropping of the image may not occur due to a resolution mismatch.

[0140]

[0141] FIG. 8b illustrates the transmission of second resolution information using RTCP in a foldable device according to one embodiment.

[0142] Referring to FIGS. 2b and FIGS. 8b, a first electronic device (210) can perform a video call with a second electronic device (220). The first electronic device (210) may be a foldable device. When a video call is initiated in a folded state (860), the video source may be an image acquired through a camera module (211). The first electronic device (210) may transmit an image frame acquired in real time through the camera module (211) to the second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) may transmit an I frame and a P frame that follows the I frame. The I frame and the P frame are image data, and the I frame and the P frame may form a group of pictures (GOP).

[0143] According to one embodiment, a screen sharing function may be executed during a video call (870). When the screen sharing function is executed, the video source may be changed from an image obtained through the camera module (211) to a captured image. For example, the screen sharing function may be executed when the first electronic device (210) is in a folded state. In the folded state, the resolution of the captured image (second resolution) may be set to a first value (e.g., 600x1280).

[0144] The first electronic device (210) can transmit to the second electronic device (220) information regarding the first value of the second resolution (e.g., 600x1280) (e.g., ratio value of 600x1280 2.13) by including it in the SDES (source description) of the RTCP packet, rather than in the RTP header or SEI message.

[0145] With the start of the screen sharing function (870), the first electronic device (210) can transmit a captured image frame to the second electronic device (220). The first electronic device (210) can transmit an I frame and a P frame that follows the I frame. The second electronic device (220) can process and display the subsequently transmitted image frames at a first value of a second resolution (e.g., 600x1280).

[0146] If the second resolution information is included in the source description (SDES) of the RTCP packet, a period (D3) in which the user can perceive a change in resolution may occur due to the RTCP period (e.g., 2 sec). In the period (D3) after the start of the screen sharing function (870) and before receiving the RTCP (875), aspect ratio distortion or corner cropping of the shared image may occur due to a resolution mismatch. In the period (E3) after receiving the RTCP (875), aspect ratio distortion or corner cropping of the shared image due to a resolution mismatch may not occur.

[0147] When the first electronic device (210) changes from a folded state to an unfolded state (880), the resolution of the captured image (second resolution) may be changed to a second value (e.g., 720x960). The first electronic device (210) may transmit to the second electronic device (220) information regarding the second value of the second resolution (e.g., 720x960) (e.g., ratio value of 1.33 of 720x960) by including it in the SDES (source description) of the RTCP packet, rather than in the RTP header or SEI message.

[0148] The first electronic device (210) can transmit a captured image frame to the second electronic device (220). The first electronic device (210) can transmit an I frame and a P frame that follows the I frame.

[0149] If the second resolution information is included in the source description (SDES) of the RTCP packet, a period (D4) in which the user can perceive a change in resolution may occur due to the RTCP period (e.g., 2 sec). In the period (D4) before receiving the RTCP (885) after the first electronic device (210) has changed from a folded state to an unfolded state (880), aspect ratio distortion or corner cropping of the shared image due to a resolution mismatch may occur. In the period (E4) after receiving the RTCP (885), aspect ratio distortion or corner cropping of the shared image due to a resolution mismatch may not occur. The second electronic device (220) can process and display the image frames transmitted thereafter as a second value of the second resolution (e.g., 720x960).

[0150] According to one embodiment, when screen sharing ends (890), the image source may be changed from a captured image to an image obtained through a camera module (211). The first electronic device (210) may transmit an image frame obtained in real time through the camera module (211) to a second electronic device (220) at a pre-negotiated first resolution (e.g., HD (720x1280)). The first electronic device (210) may transmit an I frame and a P frame that follows the I frame.

[0151] If the second resolution information is included in the source description (SDES) of the RTCP packet, a period (D5) in which the user can perceive a change in resolution may occur due to the RTCP period (e.g., 2 sec). In the period (D5) after the termination of the screen sharing function (890) and before receiving the RTCP (895), aspect ratio distortion or corner cropping of the image due to a resolution mismatch may occur. In the period (E5) after receiving the RTCP (895), aspect ratio distortion or corner cropping of the image due to a resolution mismatch may not occur.

[0152]

[0153] In an electronic device, when a first resolution (e.g., 720x1280) negotiated for video calls and a second resolution (e.g., 600x1280) for screen sharing are different, and a captured image having the second resolution is transmitted to a second electronic device (220) without transmitting information on the second resolution, distortion of the aspect ratio of the shared image or cropping of the corners of the shared image may occur due to the difference between the first resolution and the second resolution. In particular, in the case of video calls and screen sharing between a foldable device and a standard bar-shaped device, distortion of the aspect ratio may occur more significantly due to the difference in display size between the folded state and the unfolded state of the foldable device.

[0154] Although a capture method that adds a black band is used to prevent aspect ratio distortion or corner cropping of shared images, the actual area of ​​the shared screen may be reduced. Additionally, if interaction occurs due to separate user input after screen sharing, mismatched touch coordinates may cause inconvenience to the user.

[0155] The present disclosure may provide an electronic device that transmits resolution information regarding screen sharing in an RTP header or SEI message to a counterparty device performing a video call.

[0156] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0157] An electronic device according to one embodiment disclosed in this document may transmit resolution information regarding screen sharing to a counterparty device in an RTP header or SEI message when a screen sharing function is executed during a video call. Through this, the counterparty device can display the shared video without aspect ratio distortion or corner cropping.

[0158] An electronic device according to one embodiment disclosed in this document can transmit capture resolution information corresponding to various display forms of the electronic device to the other device when a screen sharing function is executed during a video call.

[0159] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0160] An electronic device (101;210) according to one embodiment may include a camera module (180;211), a display (160;212), a communication module (190), a memory (130), and at least one processor (120) including processing circuitry. The memory (130) may store instructions that cause the electronic device (101;210) to perform the following operations when executed individually or collectively by the at least one processor (120). The electronic device (101;210) may acquire first image frames of a first resolution using the camera module (180;211). The electronic device (101;210) may establish a video call by transmitting the first image frames to an external device (104;220) using the communication module (190). The electronic device (101;210) can check whether the screen sharing function is executed while the video call is in progress. If the screen sharing function is executed, the electronic device (101;210) can transmit information regarding a second resolution different from the first resolution, which is related to screen sharing, to the external device (104;220) based on the Real-Time Transport Protocol (RTP). The electronic device (101;210) can generate second image frames by capturing an image being displayed on the display (160;212) at the second resolution. The electronic device (101;210) can transmit the second image frames to the external device (104;220).

[0161] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may transmit the information to the external device (104;220) by including it in an RTP header extension.

[0162] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may include the information in a supplemental enhancement information (SEI) message and transmit it to the external device (104;220).

[0163] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may transmit the SEI message to the external device (104;220) before transmitting the I frame associated with the transmission of the second image frames.

[0164] According to one embodiment, the first resolution can be determined during the negotiation phase before connecting the video call.

[0165] According to one embodiment, the second resolution may be determined according to the aspect ratio of the width x height of the display (160;212).

[0166] According to one embodiment, information regarding the second resolution may include the ratio.

[0167] According to one embodiment, information regarding the second resolution may include the horizontal value and the vertical value.

[0168] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may resize the second image frames captured at the second resolution to the first resolution and transmit them to the external device (104;220).

[0169] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may generate the second image frames by reflecting an offset during the process of capturing an image being displayed on the display, and transmit the second image frames to the external device (104;220). The second resolution information may include information regarding the offset.

[0170] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may change the second resolution when the form of the electronic device (101;210) changes while the screen sharing function is being executed.

[0171] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may change the second resolution when the form of the display (160;212) is changed while the screen sharing function is being executed.

[0172] According to one embodiment, the electronic device (101;210) may further include a sub-display distinct from the display (160;212). When the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101;210) may change the second resolution when the display (160;212) is deactivated and the sub-display is activated while the screen sharing function is being executed.

[0173] According to one embodiment, the second resolution may be determined according to the folding state or unfolding state of the electronic device (101;210) when the electronic device (101;210) is a foldable device.

[0174] According to one embodiment, the second resolution may be determined according to the ratio of the unfolded area of ​​the display (160;212) when the display (160;212) is a rollable display.

[0175] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device (101;210) checks whether information regarding a third resolution related to screen sharing is included in a real-time transport protocol (RTP) packet received from the external device (104;220) while the video call is in progress, and if the third resolution information is included in the RTP packet, extracts a third resolution different from the first resolution from the third resolution information and renders image frames transmitted from the external device (104;220) based on the third resolution.

[0176] According to one embodiment, information regarding the third resolution may be the aspect ratio of the width x height of the display (222) included in the external device (104;220).

[0177] A method for performing a video call according to one embodiment may be performed in an electronic device (101;210). A method for performing wireless communication may include: acquiring first image frames of a first resolution using a camera module (180;211) of the electronic device (101;210); transmitting the first image frames to an external device (104;220) to establish a video call using a communication module (190) of the electronic device (101;210); checking whether a screen sharing function is executed while the video call is in progress; if the screen sharing function is executed, transmitting information regarding a second resolution different from the first resolution and related to screen sharing based on RTP (real-time transport protocol) to the external device (104;220); capturing an image displayed on the display of the electronic device at the second resolution to generate second image frames; and transmitting the second image frames to the external device (104;220).

[0178] According to one embodiment, the operation of transmitting information regarding the second resolution to the external device (104;220) may include the operation of including the information in an RTP header extension and transmitting it to the external device (104;220).

[0179] According to one embodiment, the operation of transmitting information regarding the second resolution to the external device (104;220) may include the operation of including the information in a supplemental enhancement information (SEI) message and transmitting it to the external device (104;220).

[0180] According to one embodiment, the method may include the operation of transmitting the SEI message to the external device (104;220) before transmitting the I frame associated with the transmission of the second image frames.

[0181] According to one embodiment, the first resolution can be determined during the negotiation phase before connecting the video call.

[0182] According to one embodiment, the second resolution may be determined according to the size or ratio of the display (160;212) of the electronic device (101;210).

[0183] According to one embodiment, the operation of transmitting information regarding the second resolution to the external device (104;220) may include the operation of changing the second resolution when the form of the electronic device (101;210) changes while the screen sharing function is being executed.

[0184] According to one embodiment, the operation of transmitting information regarding the second resolution to the external device (104;220) may include the operation of changing the second resolution when the form of the display (160;212) changes while the screen sharing function is being executed.

[0185]

[0186] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., by wire), wirelessly, or through a third component.

[0187] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0188] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0189] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0190] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, Camera module; display; Communication module; Memory; and It includes at least one processor comprising processing circuitry, and When the above memory is executed individually or collectively by the at least one processor, the electronic device, First image frames of a first resolution are obtained using the above camera module, and Using the above communication module, the first image frames are transmitted to an external device to establish a video call, and While conducting the above video call, check if the screen sharing function is enabled, and When the above screen sharing function is executed, information regarding a second resolution related to screen sharing and different from the first resolution is transmitted to the external device based on RTP (real-time transport protocol), and The image currently being displayed on the above display is captured at the above second resolution to generate second image frames, and An electronic device that stores instructions for transmitting the above second image frames to the external device.

2. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that includes the above information in an RTP header extension and transmits it to the external device.

3. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that includes the above information in a SEI (supplemental enhancement information) message and transmits it to the external device.

4. In paragraph 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that transmits the SEI message to the external device before transmitting the I frame associated with the transmission of the second image frames.

5. In paragraph 1, the second resolution is An electronic device characterized by being determined according to the aspect ratio of the width x height of the above display.

6. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that resizes the second image frames captured at the second resolution to the first resolution and transmits them to the external device.

7. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, In the process of capturing the image currently being displayed on the above display, the second image frames are generated by reflecting the offset, and The above second image frames are transmitted to the external device, and The information regarding the second resolution above An electronic device characterized by including information regarding the above offset.

8. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that changes the second resolution when the form of the electronic device changes while the above screen sharing function is being executed.

9. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that changes the second resolution when the form of the display changes while the above screen sharing function is being executed.

10. In Paragraph 1, It further includes a sub-display distinct from the above-mentioned display, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that changes the second resolution when the display is disabled and the sub-display is enabled while the above screen sharing function is running.

11. In paragraph 1, the second resolution is An electronic device characterized by being determined according to the folding state or unfolding state of the electronic device when the electronic device is a foldable device.

12. In paragraph 1, the second resolution is An electronic device characterized by being determined according to the ratio of the unfolded area of ​​the display when the display is a rollable display.

13. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, While conducting the above video call, check whether the RTP (real-time transport protocol) packet received from the above external device includes information regarding a third resolution related to screen sharing, and If the above RTP packet includes the above third resolution information, a third resolution different from the first resolution is extracted from the above third resolution information, and An electronic device that renders image frames transmitted from the above external device based on the third resolution.

14. In Paragraph 13, the information regarding the third resolution above An electronic device characterized by the aspect ratio of the width x height of a display included in the above external device.

15. A method for performing a video call performed on an electronic device, The operation of acquiring first image frames of a first resolution using a camera module of the electronic device; The operation of transmitting the first image frames to an external device using the communication module of the electronic device to establish a video call; An action to check whether the screen sharing function is executed while the above video call is in progress; When the above screen sharing function is executed, an operation of transmitting information regarding a second resolution different from the first resolution, which is related to screen sharing based on RTP (real-time transport protocol), to the external device; The operation of capturing an image being displayed on the display of the electronic device at the second resolution to generate second image frames; and A method comprising the operation of transmitting the above second image frames to the external device.

Images