Electronic device for displaying image and display method using same

Abstract

An electronic device according to embodiments of the present disclosure may comprise: a display module; at least one processor including a processing circuit; and a memory including a non-volatile recording medium for storing instructions. The instructions, when executed individually or collectively by at least one processor, cause the electronic device to perform one or more operations, and the one or more operations may include the operations for: acquiring a seek time and a request real time; determining a seek mode for applying an optimization mode in consideration of the ratio of times between the acquired seek time and the acquired request real time; and outputting a screen due to a requested frame on the basis of information about the time for the frame of the determined seek mode.

Description

Electronic device for displaying images and display method using the same

[0001] Various embodiments of the present disclosure relate to display technology, and more specifically, to an electronic device for displaying an image and a display method using the same.

[0002] High Dynamic Range (HDR) is a technology designed to display images similar to how users perceive objects with their own eyes by distinguishing brightness and contrast more finely. Compared to Standard Dynamic Range (SDR), HDR supports a wider color gamut and a higher luminance range, enabling the delivery of more vivid and realistic images. This allows dark areas in images or videos to be depicted as darker and bright areas as brighter, conveying realistic detail and depth.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. None of the foregoing is to be claimed as prior art related to the present disclosure, nor is it to be used to determine prior art.

[0004] Various embodiments of the present disclosure may provide an electronic device comprising: a display module; at least one processor including a processing circuit; and a memory including a non-volatile recording medium for storing instructions, wherein when the instructions are executed individually or collectively by at least one processor, the electronic device causes at least one operation to be performed, the at least one operation comprising: an operation of obtaining a seek time and a request real time; an operation of determining a seek mode for applying an optimization mode by considering the ratio of time for the obtained seek time and the obtained request real time; and an operation of outputting a screen of a requested frame based on information regarding time for a frame of the determined seek mode.

[0005] An electronic device according to embodiments of the present disclosure may include a display module, at least one processor, and a memory for storing instructions. When the above instructions are executed individually or collectively by at least one processor, the electronic device is caused to perform an operation to determine the seek mode by considering the ratio of the amount of change between consecutively acquired seek times (Δs) and the amount of change between consecutively acquired request times (Δr), wherein the amount of change between consecutively acquired seek times (Δs) includes a time difference (difference_s) between a first seek time (s1) acquired at a specific time point t1 and a second seek time (s2) acquired at a specific time point t2, and the amount of change between consecutively acquired request times (Δr) may include a time difference (difference_r) between a first request real time (r1) when the first seek time was requested at the specific time point t1 and a second request real time (r2) when the second seek time was requested at the specific time point t2. Additionally, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of determining a vector representing the speed of the seek mode using the ratio of the change amount (Δs) between the successively acquired seek times and the change amount (Δr) between the successively acquired request times; and an operation of determining a Decision Intensity representing the magnitude of the mobility of the seek mode and a Decision Direction representing the direction of movement of the seek mode using the determined vector.

[0006] A method for displaying an image according to embodiments of the present disclosure may include: an operation of obtaining a seek time and a request real time; an operation of determining a seek mode for applying an optimization mode by considering the ratio of time for the obtained seek time and the obtained request real time; and an operation of outputting a screen based on a requested frame based on information regarding time for a frame of the determined seek mode.

[0007] According to various embodiments of the present disclosure, the electronic device and display method using the same include an operation of determining the seek mode by considering the ratio of a change amount (Δs) between consecutively acquired seek times and a change amount (Δr) between consecutively acquired request times, wherein the change amount (Δs) between consecutively acquired seek times includes a time difference (difference_s) between a first seek time (s1) acquired at a specific time point t1 and a second seek time (s2) acquired at a specific time point t2, and the change amount (Δr) between consecutively acquired request times may include a time difference (difference_r) between a first request real time (r1) when the first seek time is requested at the specific time point t1 and a second request real time (r2) when the second seek time is requested at the specific time point t2.

[0008] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

[0009] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0010] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0011] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 3 is a time concept diagram according to the seek operation of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 4 is a time concept diagram according to the rendering operation of an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 5 is a conceptual diagram illustrating the seek operation process of an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram illustrating the process of seek optimization and rendering optimization of an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram showing the process of a seek mode determination unit of an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 8 is a diagram illustrating the process of a rendering optimization unit of an electronic device according to one embodiment of the present disclosure.

[0018] FIG. 9 is a time concept diagram according to an image editing operation of an electronic device according to one embodiment of the present disclosure.

[0019] FIG. 10 is a drawing illustrating the concept of editing optimization of an electronic device according to one embodiment of the present disclosure, and is a drawing explaining the concept of pen characteristics.

[0020] FIG. 11 is a conceptual diagram showing an operation for determining the editing characteristics of an electronic device according to one embodiment of the present disclosure, illustrating an image editing stroke type as an example.

[0021] FIG. 12 is a diagram illustrating the concept of editing optimization of an electronic device according to another embodiment of the present disclosure, and is a diagram schematically explaining the concept of alpha blending.

[0022] FIG. 13 is a time concept diagram according to editing optimization of an electronic device according to one embodiment of the present disclosure.

[0023] FIGS. 14a and FIGS. 14b are flowcharts illustrating the operation of an electronic device according to one embodiment of the present disclosure performing seek optimization.

[0024] FIG. 15 is a diagram illustrating a process in which an electronic device according to one embodiment of the present disclosure performs decoding.

[0025] FIGS. 16a and FIGS. 16b are flowcharts illustrating the operation of an electronic device according to one embodiment of the present disclosure performing editing optimization.

[0026] FIGS. 17a, FIGS. 17b and FIGS. 17c are drawings illustrating an example in which an electronic device according to one embodiment of the present disclosure performs image editing using seek optimization.

[0027] FIG. 18 is a time concept diagram showing the optimization effect of an electronic device according to one embodiment of the present disclosure.

[0028] Hereinafter, embodiments of this document are described in detail with reference to the drawings so that those skilled in the art can easily implement them. However, this document may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0029] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments of the present disclosure.

[0030] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0031] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0032] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0033] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0034] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or various forms of applications (146).

[0035] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0036] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof. Alternatively, an enhanced AI digital voice processing unit including an external voice processing module and a sound input module may be implemented with a logical structure, distinguished as an input module (150) or a sound (input) output module (155).

[0037] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0038] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0039] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0040] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0041] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0042] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0043] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0044] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) may be implemented, for example, as at least part of a power management integrated circuit (PMIC), or may further control the functions of the hardware by implementing an operating system (142), middleware (144), or various applications (146) or programs in relation to power management.

[0045] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery or fuel cell, and a next-generation rechargeable / sustainable alternative battery.

[0046] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, and an enhanced next-generation communication network (e.g., 5G-Adv, 6G) internet, or a long-range communication network such as a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0047] The wireless communication module (192) can support 5G, 5G-Adv, 6G, and next-generation communication technologies following the 4G network, for example, new radio access technology (NR) of a 5G system. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. Additionally, the wireless communication module (192) applying 6G artificial intelligence (AI) and machine learning (ML) technologies can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) has a Peak data rate (e.g., 20 Gbps or higher) for realizing eMBB, a loss coverage (e.g., 164 dB or lower) for realizing mMTC, or a U-plane latency (e.g., downlink (DL) and uplink (UL) each of 0) for realizing URLLC.It can support 5ms or less (or round trip 1ms or less). In addition, with the advancement of 5G-Adv, 6G, and next-generation communication technologies, newly proposed forms of AL / ML models can be utilized to perform signal processing tasks such as channel estimation, equalization, and demapping, and new image processing methods can be applied through enhanced technology. In other words, service optimization and data management / control can be performed through a more efficient network air interface.

[0048] An antenna module (197) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0050] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least a part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (101).

[0052] The electronic device (101) may provide the above result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services, for example, by using distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology. Additionally, the electronic device (101) supports 5G-Adv, 6G and next-generation communication technologies and IoT-related technologies, such as IoT security, edge computing / network slicing, energy efficiency and low power wide area (LPWA), and supports ultra-high-speed communication, low latency, and massive data transmission by utilizing terahertz and specific frequency bands for next-generation communication systems, so that IoT devices can exchange massive amounts of data in real time.

[0053] FIG. 2 is a block diagram showing the configuration of an electronic device (101) according to one embodiment of the present disclosure.

[0054] Referring to FIG. 2, the electronic device (101) may include a memory (210), a communication interface (220), a display module (230), and a processor (240).

[0055] The memory (210) can store various programs, data, instructions, etc. used in the electronic device (101). In addition, the memory (210) can store various information according to various embodiments of the present disclosure.

[0056] A memory (210) according to one example of the present disclosure may be implemented as an internal memory such as ROM (e.g., EEPROM (electrically erasable programmable read-only memory)) or RAM included in at least one processor (240), or it may be implemented as a memory separate from at least one processor (240). In this case, the memory (210) may be implemented in the form of a memory embedded in the electronic device (101) or in the form of a memory that can be attached to and detached from the electronic device (101), depending on the purpose of data storage. For example, data for operating the electronic device (101) may be stored in a memory embedded in the electronic device (101), and data for the expansion function of the electronic device (101) may be stored in a memory that can be attached to and detached from the electronic device (101).

[0057] Meanwhile, the memory embedded in the electronic device (101) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD), and the memory that is detachable from the electronic device (101) may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), or external memory that can be connected to a USB port (e.g., USB memory).

[0058] The communication interface (220) may be configured for the electronic device (101) to communicate with external devices such as an external source device or a video output device (200). The communication interface (220) may include at least one wireless communication module, at least one wired communication module, etc. Each communication module may be implemented in the form of at least one hardware chip. The wireless communication module may include at least one module among a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules. In addition, the communication interface may include Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4 th Generation), 5G(5 th It may include at least one communication chip that performs communication according to various wireless communication standards such as Generation), 5G-Adv, and 6G communication technology.

[0059] The wired communication module may include, for example, at least one of a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, a fiber optic cable, or an UWB (Ultra-Wide-Band) module. The communication interface (220) can be implemented in various forms in this way and can transmit and receive various signals by communicating with external devices.

[0060] The display module (230) refers to a configuration for displaying various content. The display module (230) may be implemented as a display including a self-emissive element or as a display including a non-emissive element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display module (230) may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc.

[0061] Meanwhile, the display module (230) can be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which multiple display modules are physically connected, etc.

[0062] At least one processor (240) controls the overall operation of the electronic device (100). Specifically, at least one processor (240) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, at least one processor (240) may be operatively connected to a memory (210), a communication interface (220), and a display module (230).

[0063] The processor (240) may be composed of one or more processors. At least one processor (240) may perform the operation of an electronic device (100) according to various embodiments by executing at least one instruction stored in memory (210). At least one processor (240) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. At least one processor (240) may control one or any combination of other components of the electronic device and may perform operations or data processing related to communication. At least one processor (240) may execute one or more programs or instructions stored in memory. For example, at least one processor may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in memory.

[0064] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0065] At least one processor (240) may be implemented as a single-core processor including one core, or as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). When at least one processor (130) is implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multi-core processor. Additionally, each of the multiple cores included in the multi-core processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure.

[0066] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0067] In the embodiments of the present disclosure, the processor may refer to a system-on-chip (SoC) in which at least one processor and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. For convenience of explanation, at least one core processor (240) will be referred to as the processor (240) below.

[0068] A processor (240) according to an embodiment of the present disclosure selectively controls multimedia editing and display output, and in particular, supports selective seek optimization operations and rendering optimization operations by checking time information for seek requests for HDR video.

[0069] In this regard, I would like to briefly explain multimedia editing and display output. First, HDR video can be classified into HDR10, which applies the same tone mapping to the entire video sequence using static metadata, and HDR10+ or ​​Dolby Vision, which apply different tone mapping to each frame using dynamic metadata. Here, HDR10 is the most widely used HDR standard and is a format natively supported by all HDR-enabled displays, while HDR10+ is an upgraded version of HDR10 that allows for more precise brightness adjustment by applying dynamic metadata to each scene. Dolby Vision is an advanced HDR technology developed by Dolby that provides higher quality video than HDR10, but requires supporting devices and content. Finally, HLG (Hybrid Log-Gamma) is an HDR format developed for broadcasting and is used to implement HDR in real-time broadcasts. When editing multimedia and outputting HDR video, seek operation and rendering operation technologies are among the key technologies for efficiently processing HDR content and enhancing playback performance. However, conventional seek operations require editing precisely to match the seek time of the frame requested by the user, which places an excessive load on the system and can lead to performance degradation due to frequent seek operations by the user. This can cause problems where the frame requested by the user is delayed even while a seek operation is in progress.Accordingly, the processor (240) according to the embodiment of the present disclosure can control more efficient seek optimization and rendering optimization operations by taking these problems into account.

[0070] Hereinafter, the seek operation and rendering operation to which the present disclosure applies will be described in detail. A seek operation is a technology that minimizes the latency occurring when rapidly moving to a specific point (seeking) during the playback of HDR video, and enables the rapid retrieval of the frame or scene desired by the user. Since HDR video handles high-capacity data, efficient seek operations are essential. For such seek operations, a keyframe (inter-frame indexing) method can be applied; this involves setting keyframes at specific points during video file encoding to support rapid movement to that location when the user seeks. Additionally, the pre-buffering method is characterized by buffering in advance during video seeking to load data for the location the user intends to move to beforehand, thereby enabling fast playback. As explained, because HDR video contains a large amount of content data, such as high resolution and color data, appropriate keyframe placement and pre-buffering are important for seek operations.

[0071] Meanwhile, editing techniques for the sought video are also a crucial factor in displaying HDR video. Rendering operations are a technology that optimizes the Graphics Processing Unit (GPU) and decoding processes when outputting HDR video in real time to play back high-quality video smoothly and quickly. Consequently, since HDR video processes a wider color space and high brightness data, the efficiency of rendering operations is critical for improving HDR system performance. As explained, GPU acceleration reduces the burden on the CPU and enables faster real-time rendering to efficiently process the complex colors, contrast ratios, and dynamic metadata of HDR video. To achieve this, both hardware and software must be optimized, and various forms of applications for external and internal operations must be utilized efficiently.

[0072] In this regard, the optional rendering optimization operation according to the embodiment of the present disclosure adaptively supports optimal rendering quality based on network conditions or device performance, and can also support adaptive streaming technology. That is, by supporting a technology that automatically adjusts the image quality of HDR content according to device performance while considering the wired / wireless communication environment of the electronic device, it supports a more efficient rendering optimization operation.

[0073] A processor (240) according to an embodiment of the present disclosure checks the received seek time and request real time for a seek request, which is a fast movement to a specific point received from an external source (e.g., a user), and determines an adaptive seek mode by comparing the time difference (amount of time change) between the seek times and the time difference (amount of time change) between the request real times. In addition, it determines whether to set a seek optimization operation by considering information related to the time of a selected frame (e.g., period information of the frame) in the determined seek mode. The processor (240) supports an optional seek optimization operation and adaptively controls a rendering optimization operation according to the determined mode to provide more efficient multimedia editing and display output. Selective control of seek operations and rendering operations provides enhanced responsiveness and rendering efficiency for incoming user seek requests.

[0074] First, FIG. 3 is a diagram illustrating the time concept according to the seek operation of an electronic device according to one embodiment of the present disclosure, and FIG. 4 is a diagram illustrating the time concept according to the rendering operation of an electronic device according to one embodiment of the present disclosure.

[0075] Referring to FIG. 3, the seeking operation resulting from the seek operation and the delay caused by video editing are illustrated. First, the seek operation refers to the process of rapidly moving to a specific time or frame in video editing, and includes a function that helps the user quickly access a desired part in video or audio editing software. Such seek operations may include time-based movement operations and frame-based movement operations. For example, time-based movement operations may include moving to a specific time on a timeline or moving to that time through touchdowns. Additionally, frame-based movement operations allow for moving frame by frame to locate an exact position and enable precise adjustment based on the frames per second (FPS). In this regard, an example according to the present disclosure briefly describes the seek operation by considering time-based movement operations. For example, a seek request (310) is requested at a specific time point (t1) at a seek (1100ms), and while a multimedia seek operation (320) and rendering (330) are performed, a seek request (310) is continuously generated from an external drag seek operation (1250ms, 1400ms, 1700ms, 1800ms, 2000ms, 2200ms, 2300ms). At this time, a seek request (2300ms) is input at a specific point in time (t2), but the frame for the seek (1100ms) of the previous specific point in time (t1) is rendered (t1100), so at the specific point in time (t2), the actual drag seek operation is to display (340) and output the frame for the seek (1100ms) which is unrelated to the requested request seek (2300ms).In such cases, even though the operation for the corresponding seek must be executed precisely in alignment with the most critical requested seek request time in a drag seek operation, continuous frame mismatches may occur due to unintended, continuous seek requests.

[0076] According to one embodiment, due to the process time of the seek operation, a gap occurs between the frame of the timestamp requested from the outside and the frame actually displayed at that time, which implies a decrease in usability. Furthermore, in a situation where a delay occurs for each seek operation process, if rendering operations are supported in a form where external interaction filtering is applied to display necessary frames and parts of the video, fewer frames are processed than the seek request input from the outside, which may cause problems such as reduced detail of the video and inability to express a seamless output. This is illustrated in FIG. 4. Referring to FIG. 4, from the video editor to the actual output, not only is the Seek Operation Delay (420) disclosed in FIG. 3 required, but a Rendering Process (430) time considering the Graphic Rendering Feature is also required. That is, even though the seek request (410) input from the outside has ended at a specific point in time (t2), the rendering delay (430) (r1100) for the previously requested seek request (1100ms) is used. From the perspective of the actual user, even though the user's drag seek request has been completed, the frame continues to be rendered and displayed with delay (r1400, r2200, r2300) (440). This can be perceived as a degradation in the rendering performance of the video editor, which leads to a decrease in user responsiveness.

[0077] According to one embodiment, when performing multimedia editing and display output, selective seek operations and rendering operations are supported to ensure maximum user responsiveness. To this end, the processor of the present disclosure determines a selective seek mode by considering the ratio of the time difference between seek times and the time difference between request real times, and displays an image by applying a selective rendering operation to the frame of the determined seek mode. Here, the speed of the seek request according to the changing seek request is checked, the direction of the drag seek is checked using the checked speed of the seek request, and the setting of the optimization mode of the seek operation is controlled by considering the checked speed and direction. In addition, rendering optimization is selectively controlled by considering the state of the electronic device, memory usage, available amount, GPU load, editing features used and the type / kind of available editing features, content management information, etc.

[0078] According to one embodiment, the concepts of Closest Seek and Closest Sync Seek are to be explained. First, Closest Seek is a method for accurately obtaining a frame at a corresponding seek time, which is a method of quickly moving to the frame closest to the time or location desired by the user. This Closest Seek is a media codec function that performs decoding through an accurate frame by utilizing an intermediate frame, such as a P-frame or B-frame, based on a previous sync frame close to the seek time, such as an I-frame. According to the above Closest Seek method, an accurate image can be obtained through the P-frame or B-frame. Meanwhile, Closest Sync Seek is a method that extracts the key frame, or I-frame, closest to the seek time requested by the user, by selecting (extracting) the I-frame of the corresponding video and using it immediately. Therefore, since the decoding logic can be completed by immediately selecting (extracting) a single frame, it has the advantage of very fast computational execution speed.

[0079] According to one embodiment, the two methods are techniques utilized when rapidly moving to a specific point in video playback. The seek operation to which the present disclosure applies causes the seek operation to operate quickly by applying the Closest Sync Seek method if a similar i-frame exists within the difference between the requested seek time and a specified reference time. Meanwhile, during drag seek, an appropriate seek mode is selected based on the time stamp difference between the requested seek time stamp and a key frame nearby. If the time stamp difference is greater than or equal to a specific threshold, the Closest Seek mode is selected; otherwise, the Instantaneous Decoder Refresh (IDR) seek mode may be selected. For a seek request resulting from a drag seek operation input by a user, the difference between the requested time stamp and the time stamp of the key frame (I-Frame) is analyzed, and if it reaches a defined threshold level, the IDR frame is displayed to perform the seek operation. On the other hand, this method has the advantage of improving seek operation performance because the seek interval for i-frames is widened; furthermore, since the aforementioned IDR frame can be decoded independently without referencing other frames, it is useful when seeking or fast playback is required. However, the closed-most seek method has the disadvantage of slow computation speed due to the characteristic of having to reference multiple frames, which results in providing delayed feedback to video editor users. Meanwhile, the closed-most sync seek method suffers from the disadvantage of reduced precision provided to the user because it continuously extracts the same frame when the user performs seek operations within the i-frame interval. To explain further, while it provides performance optimization by considering accuracy based on the difference between the seek time and the i-frame timestamp, it has the weakness of being ineffective if the user performs frequent seek operations within an i-frame interval that does not meet the aforementioned conditions.This is because the closed-seek method performs detailed decoding throughout the entire process, so while it offers good accuracy, it suffers from a performance issue where accuracy drops within the i-frame interval as frame changes are lost.

[0080] According to one embodiment, the Seek Optimization operation proposes a new seek operation that considers the advantages of closeest seek and closeest sync seek by analyzing the characteristics of the user's drag interaction and the current state of the video editor when selecting such seek frames. Here, by considering the actual time when the user requests a drag seek and the interval, speed, and direction of the i-frame, it is determined whether the current frame does not need to be sought precisely, and by performing an IDR seek based on the result of the determination, the performance overhead incurred in rendering intermediate frames is reduced, thereby enabling a rapid response to the user's request operation. In conclusion, the present disclosure is characterized by analyzing parts identified as instantaneously passing intermediate frames rather than determining the IDR seek based on the timestamp difference with the key frame, thereby not only improving overall seek performance but also simultaneously applying GPU rendering optimization to improve rendering performance according to the editing characteristics used at the time of editing based on the corresponding conditions.

[0081] FIG. 5 is a conceptual diagram illustrating the seek operation process of an electronic device according to one embodiment of the present disclosure.

[0082] Referring to FIG. 5, according to one embodiment, in the electronic device of the present disclosure, an input module (510) checks a seek request input from an external source (e.g., a user) and transmits it to a multimedia seek operation unit (520). The multimedia seek operation unit (520) determines an optional seek mode by considering information regarding the time of the seek request and time information regarding the timestamp confirmed by the input module (510), and generates a frame corresponding to the determined seek mode. To this end, one embodiment of the present disclosure defines an intermediate frame, that is, optimizes the seek time by replacing a Sync Seek, which has a relatively low load, with a Sync frame. The video editing unit (530) uses the generated frame as a background and performs final rendering by considering various forms / formats / versions of editing features added for video editing. At this time, the video editing unit (530) may consider various threads for the generation and processing of HDR video. Here, the threads include the application of various programs and applications in a multi-tasking environment where user requests and various operating systems exist during the video editing process for HDR video, and the processes include the ability to process sequentially or in parallel. It is intended to support the improvement of the performance and usability of the video editor by resizing the frames of the target content on the GPU, or by controlling the simplification of each feature stepwise and selectively when there are many editing features. At this time, the content management information may include an I-frame interval and a target frame-rate.

[0083] FIG. 6 is a diagram illustrating the concepts of seek optimization and rendering optimization of an electronic device according to one embodiment of the present disclosure.

[0084] Referring to FIG. 6, according to one embodiment, the electronic device of the present disclosure includes a seek mode selector (610), a decoding module (620), and a rendering optimizer (630). Accordingly, when a user performs a drag seek and requests a frequent seek operation, we propose an optional seek device that provides more efficient user responsiveness. At this time, we intend to explain, as an example, a case where a drag seek scenario is applied by a seek request, which is a rapid movement to a specific point received from an external source (e.g., a user).

[0085] According to one embodiment, the seek mode determination unit (610) checks the variability of the seek time by considering the change amount delta_S (Δs) between the seek time requested from the outside and the change amount delta_R (Δr) between the actual request real time when the seek was requested (occurred). The seek mode determination unit (610) selects one of the seek modes: Closest Seek, Previous Sync Seek, and Next Sync Seek, by considering the i-frame interval and the time information of the seek operation actually performed. This selects an optimal seek mode by considering the ratio of the time difference between seek times and the time difference between the actual request real time, and selects the direction of the seek operation by comparing the time difference between the seek times and the time difference between the actual request real time with a predetermined reference value. Through this, it is determined whether to apply the seek operation in an optimized mode or to operate in a normal seek mode by applying the selective seek mode and comparing it with the predetermined reference value.

[0086] According to one embodiment, the seek mode determination unit (610) transmits information regarding the selected seek mode (and information for optimization operations) to the decoding module (620). The decoding module (620) performs decoding using the information for optimization operations and transmits the information to the rendering optimization unit (630) to perform selective rendering operations on the decoded image. Based on the information, the rendering optimization unit (630), under the condition that it has entered an optimization section, predicts the overhead according to the rendering cost of the editing characteristics currently being edited by the user, and determines whether to optimize the rendering cost by reducing rendering resources in consideration of the predicted rendering cost. Afterward, according to the determination result, it outputs the final result image to which selective rendering optimization has been applied. Below, the detailed configuration of the seek mode determination unit (610), the decoding module (620), and the rendering optimization unit (630), and the operation of said configuration will be explained in more detail.

[0087] According to one embodiment, flag information regarding the determined seek mode and drag seek optimization mode is transmitted to the decoding module (620). The decoding module (620) identifies the starting point of the stream to be decoded by considering the input seek time and seek mode. Starting from the corresponding start stream, it transmits stream data to the decoder and requests decoding. In this disclosure, to minimize the processing time of the closed-list seek that proceeds after decoding, the seek time and seek mode are determined through the seek mode determination unit (610) and applied as input values ​​to the media extractor and media decoder, as an example, but this may also be applied as operating in one unit of the multimedia seek operation unit (520) of FIG. 5. That is, based on the flag information regarding the seek mode and optimization mode determined for the optimized drag seek operation determined in this disclosure, the decoding module (620) proceeds with decoding for the corresponding frame. The generated frame is converted to a color format in the video editor and passed to the rendering optimization unit (Editing Feature Optimizer, 630). This will be explained in detail with reference to Fig. 8.

[0088] FIG. 7 is a diagram illustrating the concept of a seek mode determination unit of an electronic device according to one embodiment of the present disclosure.

[0089] Referring to FIG. 7, the seek mode determination unit (610), the request analysis unit (User Request Analyzer, 710), the content analysis unit (Content Analyzer, 720), and the seek mode analysis unit (Seek mode Analyzer, 730) are included.

[0090] According to one embodiment, the seek mode determination unit (610) operates from the moment a drag seek is requested from the outside. The request analysis unit (710) checks time information regarding the input Seek Time (701) and the actual Request Real Time (703) at which the seek request occurred. At this time, the request analysis unit (710) analyzes information regarding the time difference between the Seek Time and the Request Real Time, and the time change for each time, in response to the continuously occurring seek request according to the drag seek operation. This includes analyzing the exposure time of the frame to be output due to the current user's request and determining the start time and end time according to the drag seek operation. Accordingly, the request analysis unit (710) checks each input seek time and request real time, and determines a decision vector for determining the seek mode by considering the ratio of the time difference (Δs) between the seek times and the time difference (Δr) between the request times. This is as shown in <Equation 1> below.

[0091]

[0092] According to one embodiment, the time difference between the seek times (ΔSeekTime) is a change in the seek time, and for example, includes the time difference (difference_s) between the seek time (s2) at a specific point in time t2 that was requested and the seek time (s1) at a specific point in time t1 that was previously requested. This may also include the time change amount (Δs) between the currently requested seek time and the seek time requested immediately prior. The time difference between the request real times (ΔUserRequestTime) is a change in the actual request real time at which the seek time was requested, and for example, includes the time difference (difference_r) between the actual request real time (r2) at which the seek time for a specific point in time t2 was requested and the actual request real time (r1) at which the seek time for a specific point in time t1 was requested. This may also include the time change amount (Δr) between the real time at which the current seek time was requested and the real time at which the seek time was requested immediately prior. Here, the change amount of the seek time and the change amount of the request real time may utilize an averaged value obtained by utilizing the time differences for each element acquired within each interval, taking into account interval information between the requested seek times. According to the embodiments of the present invention, various mathematical methods, functions, and modeling techniques may also be applied to measure the change amount of the seek time and the change amount of the request real time.

[0093] According to one embodiment, the request analysis unit (710) determines the speed of the seek mode and the decision vector by considering the ratio of the time difference between the seek times (ΔSeekTime) and the time difference between the request times (ΔUserRequestTime). Additionally, the request analysis unit (710) determines the Decision Intensity, which represents the magnitude of the vector and the direction of movement of the seek time, which represents the directionality, using the vector information. Here, Decision Intensity is calculated by applying a function regarding magnitude to the decision vector to determine the magnitude or strength (intensity) of the mobility of the seek mode, as shown in <Equation 2> below. In one example of the present disclosure, the absolute magnitude of the vector is calculated by applying the abs function, but the application of various functions to calculate the magnitude of the vector may be included. For example, the strength and intensity of the above decision vector can be obtained by applying the math.fabs (real numbers only) function or the distance function, which uses the distance between vectors (Euclidean distance).

[0094]

[0095] According to one embodiment, the Decision Direction is intended to determine the directionality and predicted direction of a seek operation (seek mode) using the decision vector information. At this time, the directionality of the seek operation is determined by comparing the decision vector with a predetermined reference value, and the predicted direction of the determined seek operation is estimated. This is as shown in <Equation 3> below. For example, the present disclosure sets the predetermined reference value to 0, but the reference value can be set to a different value depending on the characteristics of the video editor and the criteria for frame acquisition to ensure user responsiveness. Furthermore, since the information regarding the decision direction includes information for determining the directionality of the determined vector information, a different reference value can be set considering the editing characteristics based on a drag request and direct input values ​​received from external sources. Additionally, the invention includes the possibility of applying an enhanced decision direction estimation method using a value predetermined for the drag operation or a reference value determined considering service enhancement. Again, according to one example of the present disclosure, if the decision vector is greater than a preset reference value (e.g., 0), it is determined that it has a direction having a value greater than the preset reference value (e.g., a positive direction), and if the decision vector is smaller than the preset reference value, it is determined that it has a direction having a value smaller than the preset reference value (e.g., a negative direction).

[0096]

[0097] According to one embodiment, the request analysis unit (710) calculates a decision vector using the change in the seek time input from the outside and the change in the actual request real time whenever a seek request is input via a drag seek operation, calculates a decision intensity based on the calculated vector information, and constructs information in the decision direction by comparing the decision vector with a preset reference value. Here, the decision vector is updated in the request analysis unit (710) whenever a seek request is received. Additionally, the information is initialized by checking the last seek of the drag seek operation input from the outside. The constructed information is then transmitted to the seek mode analysis unit (730) and used to select a seek mode. The above decision vector, decision intensity, and decision direction may be updated or initialized at different intervals according to a fixed period per service or according to user settings for drag seek optimization operations. This allows for the support of seek optimization operations at different intervals by considering user patterns, the attributes of the video to be edited, and editing characteristics, in order to ensure user responsiveness and optimized rendering operations. Additionally, the request analysis unit (710) may manage and store the decision vector / intensity / direction information in the form of a single mapping table by distinguishing drag seek operations requested from the outside by specific sections / service types, or it may support efficient drag seek operations by configuring the decision vector / intensity / direction information by distinguishing it according to a fixed period / user settings in response to a specific drag seek request.

[0098] According to one embodiment, the content analysis unit (720) is a module that analyzes information about content that needs to be decoded in the current frame. The content analysis unit (720) checks the I-frame interval and analyzes the average decoding time currently being decoded in the electronic device. The I-frame interval can be checked according to the characteristics of the codec provided in the electronic device and the applied property. As another example of the present disclosure, if it is confirmed that the I-frame interval and the average decoding time according to the characteristics and performance of the codec are not supported, the content analysis unit (720) calculates the average interval obtained while performing an actual seek operation and can apply this to set the frame interval of the current electronic device. According to one embodiment, the content analysis unit (720) checks the information regarding the acquired i-frame interval and average decoding time, as well as the information regarding the decision vector, intensity, and direction acquired by the request analysis unit (710), to support the seek mode selection unit (610) in selecting a frame to operate in an optimized mode when performing a drag seek operation. Additionally, the content analysis unit (720) continuously manages the frequency at which any specific content is used in the video editor (managing cumulatively and considering a set cycle), and can organize information regarding the frequency of the content into data so that it can be reused when necessary. According to one embodiment, the content analysis unit (720) can model the usage history / preferences regarding the usage frequency and user preference of the content, that is, how often it is played and how many seeking requests are made for the content, and can analyze and manage this. Accordingly, the interval and average decoding time can be distinguished and stored in correspondence with the creation / management of a model for each content.Here, since the average decoding time is a result value that includes the current state of the current device in its meaning, the operation of the present invention can naturally be performed in accordance with the current state of the device. Additionally, the number of threads and processes of the electronic device and memory usage are also analyzed, and if the conditions for frequent use of the content are met, they can be utilized as reference information for data to be reused. Furthermore, according to one embodiment, optimization may proceed after comprehensively collecting and verifying information regarding a drag-seek operation requested by a user and information confirmed through content analysis, taking into account the information confirmed by the request analysis unit (710) and the content analysis unit (720).

[0099] According to one embodiment, a seek mode analyzer (730) determines a seek mode based on information confirmed from a request analysis unit (710) and a content analysis unit (720), and subsequently determines an optimize mode flag that indicates whether to operate in an optimized mode by considering the determined seek mode in the rendering optimization unit. The information may include at least one of the following: a decision vector, a decision intensity, a decision direction, and information regarding periods and frequencies, such as an I-frame interval, an average decoding time, or the degree of a user's request. The seek mode analyzer (730) compares the information regarding the identified decision vector, intensity, and direction with the information regarding the frame interval and sets a flag to enable (on) or disable (off) the application of the optimization mode to operate in the optimization mode. By checking the decision intensity information and the i-frame interval, if it is determined that the decision intensity is greater than the i-frame interval, and if it is determined that there is no need to decode the frame in detail, the application of the optimization mode is set (on). Then, the seek mode for the optimization mode is determined as an i-frame seek. For example, since the value currently being reviewed is measured by considering the magnitude of the decision vector calculated based on the change in the seek time and the change in the request real time, the i-frame seek is determined by considering the direction in which the seek proceeds, that is, by considering the decision direction.Among the I-frame seeks, the final seek mode is determined by considering the previous Sync Seek in the case of a negative direction and the next Sync Seek in the case of a positive direction. Meanwhile, for example, if it is confirmed that a case of frame reversal occurs, the directionality can be selectively controlled so that the seek mode does not proceed in the reverse direction when applying a drag seek operation. This can be controlled by setting the decision direction to select only the positive direction Sync Seek. In addition, the seek mode analyzer (730) determines that if the decision intensity becomes smaller than the interval of the I-frame, the frame must be decoded in detail, and decides to operate the seek mode as a closed seek, and sets the flag for the optimization mode to off.

[0100] According to one embodiment, a seek mode analyzer (730) can determine a seek mode for applying an optimization mode by comparing the average decoding time with the change amount (ΔUserRequestTime) of the requested request time. If it is confirmed that the verified average decoding time is shorter than the time difference (difference_r) between the request times input for drag seek operation, it is determined that there is no room for delay in the decoding process time for the frame, and the optimization mode flag is set to off to operate the seek mode as a closest seek. This means controlling the decoding of the correct frame through closest seek by confirming that the average decoding time provides sufficient performance compared to the time difference (difference_r) between request times, and thus the optimization mode is disabled. According to one example of the present disclosure, by selecting a seek mode considering the time difference (difference_r) between the average decoding time and the request time, and applying an optimization mode for the seek operation, more efficient drag seek operation is supported. Here, the average decoding time includes the average decoding value for frames of a predetermined value or frames of a certain length, and includes being calculated differently considering the resource usage / occupancy status of the electronic device or video editor. For example, as the decoding value varies depending on the number of applied programs or applications, the average decoding time can be variably distinguished and applied by applying different weights according to the service type, or by considering the utilization of processing resources in parallel processing programs and threads. Additionally, the average decoding time can be variably set considering the hardware or software performance of the applied decoder.This includes setting the value of the decoding time by applying different weights individually or in groups, taking into account the current state of the electronic device's processor, decoding, and editor, as well as the type of program (e.g., games or mirroring / streaming) applied to at least one multitasking service.

[0101] As described, the seek mode analyzer (730) generates flag information for the selected seek mode (705) and the determined seek operation optimization mode as an output value (707). According to an embodiment of the present disclosure, the seek mode determination unit (610) of FIG. 6 is described as being composed of a request analysis unit (710), a content analysis unit (720), and a seek mode analyzer (730) as distinct units to support an optimized seek operation; however, this may further include a processor composed of a single core process that can be configured with different logical units according to logically distinct instructions, or a separate unit added for an optimized drag seek operation.

[0102] FIG. 8 is a diagram illustrating the concept of a rendering optimization unit of an electronic device according to one embodiment of the present disclosure.

[0103] Referring to FIG. 8, the rendering optimization unit (Editing Feature Optimizer, 630) is a device that optimizes graphic rendering by considering the characteristics of video editing based on information regarding the optimization mode of the seek mode and the seek operation determined by the seek mode selection unit (Seek mode Selector). The rendering optimization unit (630) receives RGB frames (801), seek time (803), optimization mode (805), etc., as input values, and can optimize features that appear to account for a large proportion of the rendering cost among the information edited in the video editor. According to one embodiment, the rendering optimization unit (630) can perform optimization by applying graphic rendering by confirming entry information for the optimization mode based on user input based on the decision vector analyzed by the seek mode selection unit (Seek mode Selector). Accordingly, it can perform optimized rendering on the selected frame and display the rendered frame (807).

[0104] Specifically, according to one example of the present disclosure, the rendering optimization unit (630) includes a performance analyzer (810), a frame quality selector (820), and a pen quality selector (830), but the units can be added or removed in consideration of the application and threads applied to improve rendering efficiency and service. Additionally, it can be implemented in a form where a single core processor is distinguished as a logical unit, or where distinguished units are integrated under the control of a single processor. Furthermore, the rendering optimization unit (630) according to one example of the present disclosure provides a function to minimize rendering time by analyzing the editing characteristics of the video editor based on the current state of the electronic device, but rendering optimization can be supported in an adaptive form by considering rendering quality or rendering application ratio, etc., in consideration of the user's service requirements.

[0105] According to one example of the present disclosure, the rendering optimization unit (630) identifies the seek mode by utilizing an intermediate frame defined as a Sync Seek with relatively low load from the seek mode determination unit (610). Here, the seek mode determination unit (630), for example, configures the Sync Seek with low load as the interframe, that is, as a synchronous frame, to induce optimization of the seek operation. Considering this, the seek operation optimization mode according to one example of the present disclosure controls the optimization of the drag seek based on the selected interframe and optimization flag information. Since this actually provides a fast-passing frame to the user as the seek optimization mode, it means increasing the rendering speed of the frame rather than the quality of the frame. Therefore, the present disclosure supports a drag seek operation that increases responsiveness to user interaction by considering various service situations.

[0106] According to one embodiment, the performance analysis unit (810) can analyze information regarding the status of the electronic device currently used by the user and editing features used in the video editor, such as decoration, pen, tone, filter, etc. This includes the addition, deletion, or updating of various editing features depending on the performance and options of the video editor, and includes the existence of editing features that vary depending on the support of the AI ​​module according to various forms of user services and video characteristics. The performance analysis unit (810) actually applies editing features (Decoded Frame + Editing Feature) to the decoded frame to determine how much rendering cost is generated. The performance analysis unit (810) stores feature information regarding the video editor input by the user, and each feature information is used within the video editor for a certain time interval. Additionally, the performance analysis unit (810) can determine the estimated time required for the corresponding feature through a process of predicting the rendering cost. According to one embodiment, feature information is stored within the video editor for a certain time interval and can be adaptively selected, combined, and applied to be reflected during rendering. For example, if the actual rendering cost of the current electronic device is measured through a single rendering considering each feature information, the estimated time required for the corresponding feature can be determined thereafter, and a rendering can be applied in which various features are selected, combined, and variably applied considering this estimated time required.Furthermore, the embodiments of the present disclosure include the ability to verify rendering costs and additional time resulting from the application of each editing characteristic through an AI function or a specific application, or to predict different rendering costs based on the sequential or parallel combination and grouping of multiple editing characteristics for specific rendering. Additionally, through data management regarding editing characteristics added to the previous rendering cost, a new rendering cost can be utilized that takes into account newly added or changed characteristics (variable elements such as deletion, order according to new combinations, and scope of application). Here, the rendering cost calculation considering decoding time calculation can also be modified and applied, and the rendering cost can be verified by changing it according to additional resource constraints and delay factors resulting from the overall resource usage / occupancy of the terminal or other multitasking situations (such as gaming, streaming / mirroring, etc.).

[0107] FIG. 9 is a time concept diagram according to an editing operation of an electronic device according to one embodiment of the present disclosure.

[0108] Referring to FIG. 9, according to one embodiment, the rendering optimization unit can predict the delay (950, Δ_R(Rendering)) by verifying the verified rendering cost, that is, the rendering time and decoding frame extraction time for the input decoded frame and the rendered frame. For example, if the rendering time becomes slower than the decoded frame extraction time, it is determined that a delay is gradually occurring, and the performance analysis unit (e.g., 810 in FIG. 8) determines the level of the delay based on the number of decoded frames in the decoding module and adjusts the optimization level for rendering in consideration of this. Here, the optimization level can be updated in real time by considering the corresponding points in time when the applicable editing characteristics are applied. In addition, the range of the level can be variably set according to the system. For example, the performance analysis unit (810) transmits the verified level and the range of the level for optimization to the frame selection unit (820) and the pen selection unit (830) to perform rendering. According to an embodiment of the present disclosure, the rendering optimization unit is described as sharing level information and information regarding the application range of the level with the frame selection unit (820) and the pen selection unit (830), but this includes the possibility of additionally sharing and transmitting to an optimization module that is variably implemented according to the attributes of the optimization unit implemented according to editing characteristics. According to one embodiment, the performance analysis unit (810) can transmit the level and level category to the frame selection unit (820) and the pen selection unit (830) to transmit analysis results so that they can each perform optimization.Level and Level Range may indicate the degree or level of optimization being performed, and the performance analysis unit (810) may share the analysis results with the frame selection unit (820) and pen selection unit (830), which are modules performing editing optimization, so that optimization can be performed in each module by sharing information regarding the Level and Level Range indicating the degree or level of optimization being performed. It may include the degree or level of each Level and Level Range for each Feature information. Additionally, it may include the degree or level of the Level and Level Range for the grouped or whole of selected specific Feature information and combined Feature information that is grouped considering variably selected or editing characteristics for optimization efficiency. For example, regarding rendering characteristics that are adaptively selected, combined, and applied and reflected, it may include the degree and level of the Level and Level Range that can support optimal efficiency. Furthermore, considering the rendering optimization performance and cost of the electronic device, a rendering level and Level Range that select, combine, and variably apply various characteristics may be applied. Alternatively, selective rendering optimization may be performed according to criteria set under the control of the main process or the core editing unit. Or, rendering optimization may be configured with different weight information considering levels and ranges determined by the service type. As described, the rendering optimization unit ensures rendering efficiency during the editing process by sharing information for optimal rendering with each unit.

[0109] According to one embodiment, the Frame Quality Selector (820) readjusts or resizes the resolution of the frame buffer that can be controlled in the video editor based on level and level range information shared from the Performance Analysis Unit (810). That is, it includes adjusting the timing or size of data buffers, such as video streaming and audio playback, that require real-time processing to improve user responsiveness. Here, the Frame Quality Selector (820) can set a Minimum Scale, which can be changed according to the System. This is to ensure the minimum data size and resolution that must be guaranteed when outputting the video, while improving user responsiveness is also important, and this is an important factor in video service quality. Accordingly, the Frame Quality Selector (820) defines the Minimum Scale and Scaling Level / Range for optimal frame editing by considering HDR service requirements and user responsiveness, and the information can be variably set and supported for the performance of the service video and video system. Here, the frame selection unit (820) describes the level, the range of the level, the size of the frame, and the minimal scale as factors (elements) for calculating the optimal scale, for example, for the optimization of the rendering, but includes the possibility that different scale factors can be added or removed depending on the quality or performance of the video service.

[0110] According to one embodiment, the frame selection unit (820) calculates rendering optimization scaling by considering the degree of resizing of the level and the range of levels received from the performance analysis unit (810) to the size or scale of the frame. This is as shown in <Equation 4> below.

[0111]

[0112] For example, rendering costs can be reduced by checking the level range and levels to readjust the size of the frame buffers used in the video editor, thereby reducing the number of fragment pixels actually rendered. Here, a fragment pixel is a pixel on the screen treated as an individual piece or fragment, and is used in the process of calculating color, depth, texture, etc. for each pixel. As an example in the present disclosure, the above <Equation 4> sets the function by considering the frame buffer or a factor for efficient size readjustment, but this may also include the application of a modified function set by considering the individual editing characteristics configured in the editor or the combination for rendering application.

[0113] According to one embodiment, it is confirmed that a delay occurs when comparing the rendering time with the decoded frame extraction time. For example, an optimized rendering is described in which resizing is performed by setting the rendering scaling to a minimum scale of Level = 4. Here, assuming the case where Level Range = 4 and Level = 0 are set, the Level Range is defined separately, but when applying the above [Equation 4], a situation is applied where rendering optimization is not necessary. According to one embodiment, when Level = 4, the Optimized Scale is determined as the Minimum Scale, and resizing can be performed. For example, when resizing the frame buffer, since there is no scenario where the frame size becomes larger than the original, performance improvement is supported by maintaining the buffer size while reducing the area actually rendered. This is intended to provide rendering optimization by reducing the rendering area within a fixed buffer size, rather than adding memory allocation / deallocation time by deallocating and recreating the previously allocated buffer.

[0114] FIG. 10 is a drawing illustrating the concept of editing optimization of an electronic device according to one embodiment of the present disclosure, and is a drawing explaining the concept of optimization for pen characteristics.

[0115] Referring to FIG. 10, according to one embodiment, the pen selection unit (830) can adjust the editing quality by level in a manner similar to the frame selection unit (820). For example, it is assumed that there are two options for pen quality when supporting editing characteristics in the pen selection unit. For example, these are Pen Stroke Mesh Optimization and Stroke Shading Optimization. First, Pen Stroke Mesh Optimization is a readjustment of the composition of a Circle Mesh created based on consecutive points that constitute a handwritten stroke applied by the user to the video editor. For example, when applying a Circle Mesh, the more vertex points (1010) are required, the closer the shape (1020) to a circle can be drawn; conversely, if the number of vertex points is small, many edges are visible rather than circles, forming an angular polygon shape (1030). For example, since the vertex size, which is the subject of rendering, is adjusted, it is effective for controlling stroke rendering costs. In this case, since the quality of the stroke deteriorates, stroke rendering is performed by adjusting the number of vertex points only at the point of the optimization mode according to the operation for improving user responsiveness proposed in this invention. The circle mesh optimization described above is an operation of the pen selection unit according to the embodiment described above, and can be implemented by applying different rendering optimization techniques depending on the stroke shape and type used in the video editor.

[0116] FIG. 11 is a conceptual diagram illustrating an operation for determining the editing characteristics of an electronic device according to one embodiment of the present disclosure, and is a diagram illustrating a stroke type in an image editor as an example.

[0117] Referring to FIG. 11, according to one embodiment, a stroke editing characteristic in a video editor is briefly described for conceptual purposes, in which stroke types such as a normal pen (1150), a calligraphy pen (1140), a mosaic pen (1130), etc. are present. Here, performance differences in stroke rendering can occur not only during the circle mesh generation described in FIG. 10 but also in shading features. The circle mesh plays a role in maintaining the shape of the stroke by generating a basic circular mesh during the drawing process, and shading features are important for giving the stroke a three-dimensional effect or adding visual elements such as transparency and shadows. Since there are rendering costs and methods required for each stroke type, customized rendering levels and ranges corresponding to each stroke can be variably set to optimize the performance of the video editor to which the present disclosure is applied.

[0118] FIG. 12 is a conceptual diagram showing an operation for determining the editing characteristics of an electronic device according to another embodiment of the present disclosure, and is a diagram schematically explaining alpha blending characteristics.

[0119] Referring to FIG. 12, Alpha Blending, which is one of the factors affecting performance in rendering characteristics, is a technique in digital graphics that mixes two images or colors to create a transparent effect. The Alpha Channel is a value representing the transparency of a pixel, having a value (1220) between 0 (completely transparent) (1210) and 1 (completely opaque). It is an editing technique (adjusting the area of ​​1210, 1220) that uses the Alpha Channel, which represents the opacity of each pixel, to make specific parts of an image more transparent or to blend naturally with the background. This method is widely used in video editing, graphic design, game development, etc., and is one of the factors that must be considered when predicting rendering costs, as exemplified in the present disclosure. That is, if an Alpha value exists, additional rendering costs are incurred because blending must be performed with the existing drawn background area. According to one embodiment, the rendering optimization unit adjusts the rendering cost by adjusting the level of Alpha Blending by considering a determined level. As described, the rendering optimization unit according to the present disclosure may define rendering levels and ranges corresponding to each editing characteristic while supporting service requirements for video but considering user responsiveness. Alternatively, it may adaptively set rendering levels and ranges for each editing characteristic by applying different weight values ​​considering the main rendering level and range.

[0120] FIG. 13 is a time concept diagram according to Editing Feature Optimization of an electronic device according to one embodiment of the present disclosure.

[0121] Referring to FIG. 13, according to one embodiment, the rendering optimization unit reduces the time required by optimizing the rendering cost for editing characteristics by considering the configurations described in FIG. 8 to 12, each service application, editing method, etc. Accordingly, the rendering optimization unit performs rendering to minimize the rendering time (1350, Δ_O) for decoded frames and rendered frames by optimizing the identified rendering cost. At this time, by considering information regarding optimization settings for seek mode and seek operation (1310), the rendering optimization unit sets the rendering level and level range to minimize rendering that may cause delay (1350, Δ_O < Δ_R), thereby displaying the seek time requested by the user more quickly (1330), which means that the efficiency of user responsiveness of the video editor is also increased. As described, the rendering level and range for each editing characteristic can be set by applying weights with variable values ​​according to each editing characteristic, either directly or through information regarding the main defined level and range.

[0122] FIG. 14a or FIG. 14b is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure performing seek optimization.

[0123] Referring to FIG. 14a or FIG. 14b, in 1402, the process (e.g., the seek mode determination unit 610 in FIG. 6) checks whether a drag seek operation is requested from the outside. Determining whether the drag seek operation is requested or not includes checking whether a specific point in time area is requested to be sought according to the drag seek in the video editor. For example, if it is confirmed that there is a request input for a drag seek operation, it is determined to be Drag Seek on, and proceed to 1412. According to one embodiment, a drag seek includes moving playback to a desired location by dragging a user seek bar or timeline through a mouse or touch input. The input for a request for a drag seek operation may include actions such as detecting a selection of an area where a drag seek can be performed, confirming an action of performing a drag action such as a user's touch-down or touch-on on the selected area, or confirming time information regarding the moment the drag action is requested.

[0124] On the other hand, if it is determined that the Drag Seek operation is off, the process in 1404 sets the Drag Seek optimization mode to off and performs a seeking operation using the Closest Seek. Here, seeking based on the Drag Seek operation off means selecting a P-frame or B-frame based on the I-frame, which is the sync seek frame existing at the closest time or position to the input specific point in time area, and performing decoding through the correct frame using the selected intermediate frame, the P-frame or B-frame.

[0125] According to one embodiment, in 1412, a process (e.g., a request analysis unit (710 in FIG. 7)) checks the input Seek Time and the Request Real Time information where the seek is requested, and analyzes the checked Seek Time and Request Real Time information where the seek is requested. Here, the analysis of the information regarding time includes the exposure time of the frame to be output for the input user request, the start time of the drag seek operation, and the end time. In 1414, the process checks the change amount between the Seek Time transmitted as user input and the actual Request Real Time where the seek is requested in response to each seek, that is, generates a value for the respective time change amount (Delta) between the previous Seek Time and the actual Request Real Time confirmed in the previous seek request. Using the time change amount for the checked Seek Time, the time difference (Δs) between Seek Times, and the time change amount for the Request Real Time, the time difference (Δr) between Request Times, the ratio between the time differences A decision vector is determined to determine the seek mode by considering the above decision vector. Using the above decision vector, a Decision Intensity, which expresses the movement intensity of the seek time, and a Decision Direction, which expresses the movement direction of the seek time, are extracted. For example, Equations 1 to 3 have been explained to calculate values ​​for identifying the characteristics of the seek operation, but different functions and modeling techniques can be applied in various forms to generate the speed, intensity, and directionality of the seek operation.

[0126] According to one embodiment, in 1422, a process (e.g., a content analysis unit (720 in FIG. 7)) analyzes information about the content to be decoded. It analyzes the content information of the seek requested frame by considering value(s) for identifying the characteristics of the seek operation. For example, the interval of an I-frame written in the content file is checked and used as a criterion for determining the seek operation optimization mode. If the interval of an I-frame cannot be checked from the corresponding file, in 1424, the process may proceed to a process of analyzing the average I-frame interval according to one embodiment of the present disclosure. In the analysis process in 1424, it is determined that a drag seek request is continuously received according to the operation of the seek operation, and thus a situation arises where the average frame interval can be obtained through the seek time according to the seek request. The average interval obtained in this way can be applied by setting the frame interval of the current electronic device. Once the content file is analyzed, you can proceed to the next step.

[0127] According to one embodiment, in the above 1422, the process (e.g., the seek mode analysis unit (730 in FIG. 7)) determines whether a seek mode can be determined for a seek operation optimization mode by checking whether an i-frame interval exists, and proceeds to 1432 (A in FIG. 14a, 1432 in FIG. 14b). In 1432, the processor determines the seek mode by considering the confirmed i-frame interval. In 1434, the processor can determine a frame to operate in optimization mode by using information regarding the i-frame interval and average decoding time, as well as information regarding the decision vector, intensity, and direction. Accordingly, by comparing the average decoding time and the change amount of the requested request time (ΔUserRequestTime), for example, it can be checked whether the average decoding time is smaller than the time change amount (Delta) of the request time (ΔUserRequestTime) currently requested by the user. If the time change amount (ΔUserRequestTime) is small, it is determined that frame decoding is performed faster than the user request time (in the case of No B of 1434 in FIG. 14b), so the optimization mode setting (on) is not necessary, and thus the process proceeds to the closed-seek mode (1404) for accurate frame decoding. In 1404, the process (e.g., the seek mode analysis unit (730 in FIG. 7)) controls the drag-scroll optimization mode to off and performs the closed-seek mode. On the other hand, if the average decoding time is greater than the time change amount (Delta) of the current user request time (ΔUserRequestTime), for example, if it is confirmed that the average decoding time is slower than the request time (ΔUserRequestTime) according to the seek request, it is determined that the decoding is delayed compared to the user request, and the process can proceed to 1442.

[0128] According to one embodiment, in 1442, the process (e.g., the seek mode analysis unit (730 in FIG. 7)) compares the intensity of the identified i-frame interval and the seek time (I-frame Interval vs. Decision Intensity) to determine whether the decision intensity is greater than the i-frame interval. Here, if the decision intensity is greater than the i-frame interval, it is determined that there is no need to extract an exact frame between i-frames, and accordingly, it is determined that an intermediate frame of the intermediate process performing the drag is requested. According to one embodiment, the application of Optimize mode can be set to On, thereby determining the seek mode as an I-frame Seek. According to one embodiment, a Sync Seek, which has a relatively low load, can be determined as the i-frame using the intermediate frame, and this can be used to operate in Drag Seek Optimization mode. At this time, when performing an i-frame seek, since the closest i-frame is retrieved, a frame inversion phenomenon may occur where a previously displayed frame reappears in a direction that does not match the direction of movement. To prevent this inversion phenomenon, the process can be performed with 1444.

[0129] According to one embodiment, at 1444, the processor checks whether the decision vector is greater than a preset reference value. The decision direction is determined using the decision vector and the preset value, or a reference value determined by considering service enhancement. The direction of the seek operation is determined based on the result of comparison with the reference value. At this time, if the decision vector is greater than the preset reference value (e.g., 0), it is determined that it has a direction (e.g., a positive direction) that has a value greater than the reference value, and the process proceeds to 1452 in the normal seek time direction to extract and retrieve the subsequent frame through the next sync seek. According to one embodiment, when extracting an I-frame seek, the closest I-frame is retrieved, so an inversion phenomenon may occur in which a previously shown frame is displayed again in a way that does not match the direction of movement. To prevent this phenomenon, the direction of progress is checked based on the decision vector; if it is in the normal seek time direction (e.g., greater than the reference value), the next frame is acquired to retrieve the frame, while if it is in the opposite direction (e.g., less than the reference value), the previous frame is checked to prevent the inversion phenomenon.

[0130] According to one embodiment, if the processor in 1454 determines that the opposite case is true, that is, if the decision vector is smaller than a preset reference value, it determines the direction to have a value smaller than the reference value (for example, a negative direction) and can retrieve the previous frame through a previous sync seek.

[0131] According to an embodiment of the present disclosure, the processor can determine information regarding the decision vector, intensity, and direction of the seek operation through the respective changes in the seek time and request real time according to the seek operation, and determine a selective frame to operate in an optimization mode using the confirmed information regarding the seek operation. In addition, by considering the comparison and analysis process and the predetermined condition process with information regarding the i-frame interval and average decoding time, the processor ultimately determines the seek optimization operation, thereby providing the effect of reducing the decoding time through the i-frame seek mode. Furthermore, by sharing the selected seek mode and the on / off setting of the optimization flag for the application of the seek mode during video rendering, the processor provides the advantage of inducing not only the optimization of the drag seek operation but also the rendering optimization operation for the selected video.

[0132] FIG. 15 is a conceptual diagram illustrating the operation of an electronic device performing decoding according to one embodiment of the present disclosure.

[0133] Referring to FIG. 15, the decoding module (1500) is a logic for decoding content and transmits a seek mode to the extractor in the decoding module to determine whether to perform i-frame seek or to perform real seek through P-frame or B-frame. By transmitting the seek time, the encoded frame can be decoded according to the seek mode and one frame can be transmitted.

[0134] According to one embodiment, it may include a setting unit (1510) that initializes a decoding module and configures it by analyzing a frame, and an image processing unit (1520) that stores the decoded image and converts it into image data. The setting unit (1510) may perform decoding setup / configuration (1511), module initialization (1513), and error detection and handling (1515). The module setup / configuration (1511) operation is a process of setting / completing configuration information and settings necessary for the decoding module (1500) to operate normally, and may perform processes such as decoder parameter setting, stream information parsing, and hardware acceleration setting. The module initialization (1513) operation is a process of preparing for the execution of the decoder module, and may include hardware and software environment settings and PU / GPU resource initialization. Additionally, the module setup / configuration (Setup / Configuration, 1511) operation and module initialization (Initialization, 1513) share signals for encoding start (1501) and end (1503) to allow the image processing unit (1520) to prepare an input buffer, create a buffer to receive encoded image data, and prepare an output buffer to allocate memory space to store the decoded image, and support the conversion / reconfiguration of the initialization and setup configuration of the decoded image by considering the operation of flushing (1521) and running (1523) the image in the image buffer and memory.Accordingly, during the video decoding operation, when the data stream unit (1525) decompresses the encoded input data and converts it into original video data, the encoded data stream can be controlled to be input or output again to the decoder module (1500) to generate the corresponding frame normally. At this time, through the error detection and processing operation (1515), problems occurring during the decoding process are detected and processed, and the module setup / configuration (1511) operation and module initialization (1513) are reset or controlled to operate normally by considering the error information (1507), and if necessary, a notification can be provided to the user. The termination operation (1530) is a process of cleaning up and terminating the resources of the decoder module (1500), and the decoding process can be terminated by receiving a signal (1509) to release the buffer and memory within the video processing unit (1520) and return the used hardware / software resources. According to one embodiment, a buffer pool can be formed to perform a sequencing operation so that the decoding input and output of the decoding module can proceed asynchronously. For example, if there is a buffer that can be used as an input buffer, its availability is indicated, and the decoding operation can be performed efficiently by internally considering this. The buffer that has completed decoding is accumulated in the output queue, and when the user releases the buffer (1530) after using the corresponding frame, the buffer is subsequently used as an input buffer, allowing the buffer pool to be used efficiently by continuously using the decoded buffer in the buffer pool in an input form or by managing it in an output form. According to one embodiment of the present disclosure, the decoded output generated by the decoding module can be reflected in graphic rendering and used for rendering optimization operations.

[0135] FIG. 16a or FIG. 16b is a flowchart illustrating the operation of an electronic device according to an embodiment of the present disclosure performing editing feature optimization. The editing feature optimization device illustrated in FIG. 16a or FIG. 16b is characterized by defining a rendering level and a rendering range for rendering optimization when a delay may occur in the rendering time of the current electronic device due to the complexity of editing features added by a user, and further optimizing the frame buffer size and stroke precision in consideration of the defined rendering level to reduce the graphic rendering cost. That is, it includes analyzing the rendering cost and operating in an optimization mode according to the rendering level in consideration of editing features.

[0136] Referring to FIG. 16a or FIG. 16b, in 1602, a process (e.g., the rendering optimization unit (630) of FIG. 6) checks information regarding the seek mode optimization determined through the seek mode determination unit (e.g., the seek mode selector (610) of FIG. 6). In 1602, the processor can check information regarding the transmitted RGB frame, seek time, and flag settings for the optimization mode. In 1612, the process checks whether it should operate in the optimization mode according to the optimization mode settings. If the optimization mode is set, in 1612, the process predicts the current rendering cost. Here, by checking information such as the priority of the analyzed rendering cost and the weight for optimization for the rendering group per service, it checks whether an optimized rendering cost exists. Here, it checks the estimated time required when rendering is operated at the requested seek time due to editing characteristics added by the user. At this time, if the corresponding time is not calculated, proceed to 1614, and the process performs normal rendering without rendering optimization, and then analyzes the rendering cost at 1616. Here, the rendering cost analysis calculates the rendering time for each editing characteristic applied by the user, which implies that the current state of the electronic device has the advantage of being able to respond to rendering in real time, and is a performance prediction device possible only in video editors.

[0137] According to one embodiment, if the process in 1622 confirms that an analyzed rendering cost exists, it proceeds to 1632 to check the rendering time and the decoding frame extraction time to measure the latency relative to the decoding time. If it is confirmed in 1634 that the analyzed rendering time is greater than the decoding time—that is, if it is confirmed that the decoding time takes longer—the process proceeds to 1640. If it is confirmed in 1634 that the analyzed rendering time is not greater than the decoding time, the process proceeds to 1614. In 1640, the level of latency is determined in 1622 by considering the latency, and the optimization level for rendering is determined by considering this. That is, the rendering level for optimization and the range of the level are extracted by calculating at the level where latency occurs. Here, the criteria for calculating the level and the range of the level may vary dynamically depending on the situation of the video editor. Here, the decoding time or rendering cost may be set as a fixed value, but may further include being set variably depending on the resource situation of the electronic device. Here, when the processor of the electronic device is used with a threshold value higher than a predetermined resource share, the parameters for the above-mentioned editing optimization may be set as variable parameters with a constant weight for optimization operation. For example, if the level range is 4 and the current level is calculated to be 4, it is determined that the situation requires the most optimization in 4 stages.

[0138] According to one embodiment of the present disclosure, when applying optimization to frame quality using values ​​for the level and the application range of the level detected for the optimization mode, resolution readjustment or frame resizing is performed by considering the level of the resolution of the frame buffer at 1642. Or, if pen characteristic optimization is applied, the stroke mesh complexity at 1644 is reduced by considering the level of the extracted stroke mesh complexity. Or, if stroke alpha blending characteristics are applied, the rendering cost is optimized by reducing the sensitivity of alpha blending according to the level of the blending intensity at 1646. According to one embodiment, the frame buffer size is adjusted by considering the level of the resolution of the frame buffer at the frame selection unit (820), the stroke mesh complexity is reduced according to the level at the pen selection unit (830), and if necessary, the stroke alpha blending intensity is reduced according to the level to finally proceed with rendering. Rendering cost analysis can be performed even after this. If rendering occurs again in the next cycle and performance improvements are required even after applying the corresponding level, the rendering cost can be ultimately optimized by repeating the process of recalculating the level and level range based on an analysis of the difference between the analyzed rendering cost and the decoding time.

[0139] In this disclosure, frame selection, panning, and alpha blending are described as examples; however, rendering optimization reflecting different characteristics may be applied depending on the video service quality and editing characteristics applied to the supported video editor. Subsequently, rendering cost analysis may be performed variably according to the editing characteristics or the frequency to ensure user improvement. Additionally, optimized rendering operations are applied according to user preferences and frequency, and selective rendering optimization may be performed according to the set seek optimization operations. Here, if performance improvement for optimization is required even after applying the corresponding rendering level, the difference between the analyzed rendering cost and decoding time can be analyzed to recalculate the level and the range of application for the optimization, and the process can be repeated to finally optimize the rendering cost. Furthermore, since the analyzed rendering cost and decoding time have the characteristic that they may vary depending on the resource conditions of the electronic device used, the values ​​for rendering optimization may be set and applied to increase accuracy based on iterative calculations, AI / ML modeling, and video editing performance.

[0140] FIG. 17a, FIG. 17b, or FIG. 17c is a drawing illustrating an example in which an electronic device according to one embodiment of the present disclosure performs image editing using seek optimization.

[0141] Referring to FIG. 17a, a drag seek operation according to one embodiment of the present disclosure is a case (1711) in which content is sought continuously using a preview bar or a user drag interaction within a video editor. Here, confirming that a drag seek has started from the outside, that is, detecting that an entry for a drag seek exists, is, for example, when a touch down (1713) occurs on the user's preview bar. When such an operation begins, an operation to perform a seek operation is performed. Here, even if the user commands the device to enter an optimization mode, such an optimization mode may not always be active. This allows for the adaptive application of an optimization operation according to service performance and level, taking into account conditions that are prioritized according to the characteristics of the electronic device to which the present disclosure applies and service quality, etc.

[0142] Referring to FIG. 17b, confirming from the outside that the drag seek has ended—that is, detecting that completion for the drag seek exists—is, for example, when a user touches up (1723). Accordingly, when the drag seek operation is stopped through the touch up (1721), the optimization device no longer operates. Here, the seek operation is performed even at the time of termination, so that the last requested seek time is not an intermediate frame, but rather the optimization mode is disabled, allowing the output of a frame of accurate time with good service quality. Additionally, even if the optimization operation is not performed due to the optimization mode being disabled, the information analyzed during the drag seek operation can be managed, stored, and maintained. This includes checking information on the drag seek optimization operation at a set interval and applying the optimization of the drag operation that meets specific conditions as a reference, thereby enabling faster and more efficient drag seek operations.

[0143] Referring to FIG. 17c, the diagram shows a touch move interaction performed through dragging (1731, 1741) in a touchdown state (1735, 1745). The interaction according to the present embodiment includes extracting the direction (positive direction (1735), negative direction (1745)) and movement intensity of the drag seek using the input seek time, the interval of the i-frame, etc. Accordingly, it provides the advantage of determining whether the current drag seek is operating quickly in response to a user's request, and selectively generating an intermediate frame to support faster responsiveness rather than a frame with high decoding complexity that guarantees service quality, for the frame extracted due to the request of the drag seek. The drag seek operation of the electronic device according to the present embodiment extracts a selective seek mode by considering the amount of change of time elements related to the drag seek and the interval of the reference frame for the seek mode, and performs an optimized seek operation and rendering based on the information. Accordingly, the seek optimization operation and rendering operation according to the present disclosure have the advantage of resolving the problem of decoding delay, which requires always outputting an accurate frame, without considering whether existing seek requests occur frequently or are input slowly. Accordingly, it has the advantage of providing an improved dress seek operation service by resolving the degradation of user responsiveness when using drag seek.

[0144] FIG. 18 is a diagram showing the optimization effect of an electronic device according to one embodiment of the present disclosure.

[0145] Referring to FIG. 18, for a user's requested seek request operation (1810), as an example, a state in which a seek request is continuously generated from an external drag seek operation (1000ms, 1100ms, 1400ms, 1700ms, 1800ms, 2000ms, 2200ms, 2300ms). For example, it can be seen that the time (O_t1) (1870) taken to output to the display (1840) of the corresponding frame, where the multimedia seek operation (1820) and rendering (1830) for the seek request (1100ms, 1810) at a specific time point (t1, 1811) is reduced to before the multimedia seek operation (1820) for the seek request (1400ms) at a specific time point (t2, 1813), and the time (O_t2) (1880) taken to output to the display (1840) of the corresponding frame, where the multimedia seek operation (1820) and rendering (1830) for the seek request (2300ms) at a specific time point (t3, 1821) is performed, is also reduced. Furthermore, as the execution time for requests is reduced, the number of frames processing user-requested seeks increases, allowing frames to be updated more efficiently. This provides the advantage of faster response times for seek operations requested by users while performing drag seeks.

[0146] According to one embodiment, in supporting drag seek operations, instead of lowering frame quality and accuracy due to operation in an unconditional optimization mode, the system adaptively selects an accurate frame or a frame for optimization mode by considering the i-frame interval and decoding time, thereby supporting the maintenance of the original quality when the drag seek operation ends. Through this, service quality can be guaranteed so that no quality degradation issues occur when the seek operation ends.

[0147] According to one embodiment, the electronic device may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the aforementioned devices.

[0148] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0149] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0150] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0151] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0152] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0153] An electronic device according to embodiments of the present disclosure comprises a display module, at least one processor including a processing circuit, and a memory including a non-volatile recording medium for storing instructions, wherein when the instructions are executed individually or collectively by at least one processor, the electronic device causes at least one operation to be performed, the at least one operation may cause the electronic device to perform at least one operation, the operation of obtaining a seek time and a request real time, the operation of determining a seek mode for applying an optimization mode by considering the ratio of time for the obtained seek time and the obtained request real time, and the operation of outputting a screen by a requested frame based on information about time for a frame of the determined seek mode.

[0154] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device is caused to perform an operation to determine the seek mode by considering the ratio of the amount of change between consecutively acquired seek times (Δs) and the amount of change between consecutively acquired request times (Δr), wherein the amount of change between consecutively acquired seek times (Δs) includes a time difference (difference_s) between a first seek time (s1) acquired at a specific time point t1 and a second seek time (s2) acquired at a specific time point t2, and the amount of change between consecutively acquired request times (Δr) may include a time difference (difference_r) between a first actual request real time (r1) when the first seek time was requested at the specific time point t1 and a second actual request real time (r2) when the second seek time was requested at the specific time point t2.

[0155] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of determining a vector representing the speed of the seek mode using the ratio of the change amount (Δs) between the successively acquired seek times and the change amount (Δr) between the successively acquired request times; and an operation of determining a Decision Intensity representing the magnitude of the mobility of the seek mode and a Decision Direction representing the direction of movement of the seek mode using the determined vector.

[0156] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of updating a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode whenever a seek request is received; and an operation of initializing a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode in response to the last seek request.

[0157] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to: calculate a vector representing the speed of the seek mode using the ratio of the change amount (Δs) between the successively acquired seek times and the change amount (Δr) between the successively acquired request times using the following <Equation 5>.

[0158]

[0159] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to: calculate the Decision Intensity, which represents the magnitude of the mobility of the seek mode using the vector, using the following <Equation 6>.

[0160]

[0161] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to: calculate a Decision Direction indicating the direction of movement of the seek mode using the determined vector and a predetermined reference value using the following <Equation 7>.

[0162]

[0163] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of updating a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode whenever a seek request is received; and an operation of initializing a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode in response to the last seek request.

[0164] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of determining an interval for the period of an I-frame; an operation of determining an average decoding time of the electronic device; and an operation of determining a seek mode by comparing the determined interval and the determined average decoding time.

[0165] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation to determine the seek mode by comparing the identified interval and the identified average decoding time according to a predetermined standard with a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode.

[0166] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: the operation of setting the optimization mode to ON in response to the intensity being greater than the interval; and the operation of determining the seek mode as an I-frame seek for the optimization mode.

[0167] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining the seek mode as the next sync seek by determining that the direction of movement of the seek mode has a positive direction if the vector is greater than a predetermined reference value.

[0168] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining the seek mode as a previous sync seek by determining that the direction of movement of the seek mode has a negative direction if the vector is smaller than the predetermined reference value.

[0169] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining a rendering optimization mode based on the rendering time and the decoding frame extraction time for the decoded frame and the rendered frame.

[0170] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation to determine the optimization mode by adjusting the range of the rendering optimization level according to the characteristics of each image editing and the optimization level for rendering based on the determined seek mode.

[0171] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of determining a rendering delay level by considering the number of decoded frames; an operation of adjusting an optimization level for rendering by considering the determined delay level; and an operation of determining the optimization mode by adjusting the range of the rendering optimization level according to the characteristics of each video editing.

[0172] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device causes the electronic device to perform: an operation of variably setting the characteristics of each video editing according to the performance of the video editor and the quality of service corresponding to the user's requirements; and an operation of selectively displaying a video with rendering editing characteristics applied, taking into account the optimization level adjusted for each characteristic of each video editing and the range of the optimization level, wherein each characteristic of each video editing may include at least one of a pen characteristic, a stroke type, and an alpha blending.

[0173] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to: calculate rendering scaling, to which rendering editing characteristics are selectively applied in consideration of the optimization level and the range of the optimization level, using the following <Equation 8>.

[0174]

[0175] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform the operation of applying an optimized seek operation according to the determined seek mode and the optimization mode; and the operation of displaying an image with an optimized rendering operation applied by applying an optimization level and a range of optimization levels adjusted according to the characteristics of each image editing.

[0176] According to one example, the electronic device may include a display module. The electronic device may include at least one processor including a processing circuit. The electronic device may include a memory including a non-volatile recording medium for storing instructions. When the instructions are executed individually or collectively by at least one processor, the electronic device may cause at least one operation to be performed. The at least one operation may include: an operation of obtaining a seek time and a request real time; an operation of determining a seek mode for applying an optimization mode by considering the ratio of time for the obtained seek time and the obtained request real time; and an operation of causing a screen of a requested frame to be output based on information regarding time for a frame of the determined seek mode.

[0177] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation to determine the seek mode by considering the ratio of the change amount (Δs) between consecutively acquired seek times and the change amount (Δr) between consecutively acquired request times.

[0178] According to one example, the amount of change (Δs) between the consecutively obtained seek times may include the time difference (difference_s) between the first seek time () obtained at a specific time point t1 and the second seek time (s2) obtained at a specific time point t2.

[0179] According to one example, the amount of change (Δr) between the consecutively obtained request times may include the time difference (difference_r) between the first actual request real time (r1) at which the first seek time is requested at the specific time point t1 and the second actual request real time (r2) at which the second seek time is requested at the specific time point t2.

[0180] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining a vector representing the speed of the seek mode using the ratio of the change amount (Δs) between the successively acquired seek times and the change amount (Δr) between the successively acquired request times.

[0181] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining a Decision Intensity representing the magnitude of the mobility of the seek mode and a Decision Direction representing the direction of movement of the seek mode using the determined vector.

[0182] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of updating a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode whenever a seek request is received.

[0183] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of initializing a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode in response to the last seek request.

[0184] According to one example, when the above instructions are executed individually or collectively by at least one processor, they may cause the electronic device to perform an operation of checking the interval for the period of an I-frame (I-frame Interval).

[0185] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation to check the average decoding time of the electronic device.

[0186] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining the seek mode by comparing the identified interval and the identified average decoding time.

[0187] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation to determine the seek mode by comparing the identified interval and the identified average decoding time according to a predetermined standard with a vector representing the speed of the seek mode, a decision intensity representing the magnitude of the mobility of the seek mode, and a decision direction representing the direction of movement of the seek mode.

[0188] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform the operation of setting the optimization mode on in response to the intensity being greater than the interval.

[0189] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining the seek mode as an I-frame seek for the optimization mode.

[0190] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of setting the optimization mode off in response to the intensity being smaller than the interval.

[0191] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining the seek mode as a closed seek to release the optimization mode.

[0192] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining the seek mode as the next sync seek by determining that the direction of movement of the seek mode has a positive direction if the vector is greater than a predetermined reference value.

[0193] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining the seek mode as a previous sync seek by determining that the direction of movement of the seek mode has a negative direction if the vector is smaller than the predetermined reference value.

[0194] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of determining a rendering optimization mode based on the rendering time and decoding frame extraction time for the decoded frame and the rendered frame.

[0195] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation to determine the optimization mode by adjusting the range of the optimization level for rendering and the rendering optimization level according to the characteristics of each image editing based on the determined seek mode.

[0196] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of checking a rendering delay level by taking into account the number of decoded frames.

[0197] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of adjusting the optimization level for rendering in consideration of the identified delay level.

[0198] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation to determine the optimization mode by adjusting the range of rendering optimization levels for each characteristic of video editing.

[0199] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of variably setting the characteristics of each video editing according to the performance of the video editor and the quality of service corresponding to the user's requirements.

[0200] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of selectively displaying an image with rendering editing characteristics applied, taking into account the optimization level and the range of the optimization level adjusted for each image editing characteristic.

[0201] According to one example, each of the above video editing characteristics may include at least one of pen characteristics, stroke type, and alpha blending.

[0202] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of applying an optimized seek operation according to the determined seek mode and the optimization mode.

[0203] According to one example, when the above instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of displaying an image with an optimized rendering operation applied by applying an optimization level and a range of optimization levels adjusted for each image editing characteristic.

[0204] According to one example, a method for displaying an image may be included. The method for displaying the image may include at least one operation. The at least one operation may include an operation of obtaining a seek time and a request real time. The at least one operation may include an operation of determining a seek mode for applying an optimization mode by considering the ratio of time between the obtained seek time and the obtained request real time. The at least one operation may include an operation of outputting a screen based on a requested frame based on information regarding time for a frame of the determined seek mode.

[0205] According to one example, the at least one operation may include an operation to determine the seek mode by considering the ratio of the amount of change (Δs) between consecutively acquired seek times and the amount of change (Δr) between consecutively acquired request times.

[0206] According to one example, the at least one operation may include a time difference (difference_s) between a first seek time (s1) obtained at a specific time point t1 and a second seek time (s2) obtained at a specific time point t2, as a change amount (Δs) between consecutively obtained seek times.

[0207] According to one example, the at least one operation may include a time difference (difference_r) between the first request real time (r1) at which the first seek time is requested at the specific time point t1 and the second request real time (r2) at which the second seek time is requested at the specific time point t2, as a change amount (Δr) between consecutively acquired request times.

[0208] According to one example, the at least one operation may include an operation to determine a vector representing the speed of the seek mode using the ratio of the amount of change (Δs) between consecutively acquired seek times and the amount of change (Δr) between consecutively acquired request times.

[0209] According to one example, the at least one operation may include an operation to determine a Decision Intensity representing the magnitude of the mobility of the seek mode and a Decision Direction representing the direction of movement of the seek mode using a determined vector.

[0210] According to one example, the at least one operation may include updating a vector representing the velocity of the seek mode, a Decision Intensity representing the magnitude of the mobility of the seek mode, and a Decision Direction representing the direction of movement of the seek mode whenever a seek request is received.

[0211] According to one example, the at least one operation may include initializing a vector representing the velocity of the seek mode, a Decision Intensity representing the magnitude of the mobility of the seek mode, and a Decision Direction representing the direction of movement of the seek mode in response to the last seek request.

[0212] According to one example, the above at least one operation may include an operation to check the interval for the period of an I-frame (I-frame Interval).

[0213] According to one example, the above at least one operation may include an operation to check the average decoding time.

[0214] According to one example, the at least one operation may include an operation to determine a seek mode by comparing a confirmed interval and a confirmed average decoding time.

[0215] According to one example, the at least one operation may include comparing a identified interval and an identified average decoding time with a vector representing the speed of the seek mode, a Decision Intensity representing the magnitude of the mobility of the seek mode, and a Decision Direction representing the direction of movement of the seek mode according to a defined standard.

[0216] According to one example, the at least one operation may include an operation to turn on the optimization mode in response to the intensity being greater than the interval.

[0217] According to one example, the at least one operation may include an operation of determining the seek mode as an I-frame seek for an optimization mode.

[0218] According to one example, the at least one operation may include an operation to turn off the optimization mode in response to the intensity being smaller than the interval.

[0219] According to one example, the above at least one operation may include an operation of determining the seek mode as a closed seek to release the optimization mode.

[0220] According to one example, the above at least one operation may include determining the seek mode as the next sync seek by determining that the movement direction of the seek mode has a positive direction when the vector is greater than a predetermined reference value.

[0221] According to one example, the above at least one operation may include determining the seek mode as a previous sync seek by determining that the movement direction of the seek mode has a negative direction if the vector is smaller than the predetermined reference value.

[0222] According to one example, the at least one operation may include an operation to determine a rendering optimization mode based on the rendering time and decoding frame extraction time for the decoded frame and the rendered frame.

[0223] According to one example, the above at least one operation may include an operation to determine an optimization mode by adjusting the range of the rendering optimization level according to the characteristics of each video edit and the optimization level for rendering based on the determined seek mode.

[0224] According to one example, the above at least one operation may include an operation to check the rendering delay level by considering the number of decoded frames.

[0225] According to one example, the at least one operation may include an operation to adjust the optimization level for rendering in consideration of the identified delay level.

[0226] According to one example, the at least one operation may include an operation to determine the optimization mode by adjusting the range of the rendering optimization level for each characteristic of the video editing.

[0227] According to one example, the at least one operation may include an operation to variably set the characteristics of each video edit according to the performance of the video editor and the quality of service corresponding to the user's requirements.

[0228] According to one example, the at least one operation may include an operation of outputting a video to which rendering editing characteristics are applied, taking into account the optimization level adjusted for each video editing characteristic and the range of the optimization level.

[0229] According to one example, the above at least one operation may include at least one of a pen characteristic, a stroke type, and alpha blending as a characteristic of each video editing.

[0230] According to one example, the at least one operation may include applying an optimized seek operation according to a determined seek mode and an optimization mode.

[0231] According to one example, the above at least one operation may include an operation of outputting a video with an optimized rendering operation applied by applying an optimization level and a range of optimization levels adjusted for each characteristic of video editing.

[0232] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a display device (e.g., TV, monitor, light projection device), a portable communication device (e.g., smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0233] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0234] As used in various embodiments of this document, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0235] Various embodiments of the present document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory (430)) readable by a machine (e.g., an image projection device (100)). For example, a processor (e.g., a processor (410)) of the machine (e.g., an image projection device (100)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0236] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0237] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, Display module; At least one processor including a processing circuit; and It includes memory including a non-volatile recording medium that stores instructions, When the above instructions are executed individually or collectively by at least one processor, they cause the electronic device to perform at least one operation, and the at least one operation is: An operation to acquire Seek Time and Request Real Time; An operation to determine a seek mode for applying an optimization mode by considering the ratio of the time for the above-mentioned obtained seek time and the above-mentioned obtained request real time; and An operation to output a screen based on a requested frame based on time information for the frame of the above-determined seek mode, An electronic device including 2. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Causing to perform an operation to determine the seek mode by considering the ratio of the amount of change between consecutively acquired seek times (Δs) and the amount of change between consecutively acquired request times (Δr), and The change amount (Δs) between the above consecutively acquired seek times includes the time difference (difference_s) between the first seek time (s1) acquired at a specific time point t1 and the second seek time (s2) acquired at a specific time point t2, and An electronic device wherein the amount of change (Δr) between the consecutively acquired request times includes the time difference (difference_r) between the first request real time (r1) at which the first seek time is requested at the specific time point t1 and the second request real time (r2) at which the second seek time is requested at the specific time point t2.

3. In Paragraph 2, When the above instructions are executed individually or collectively by at least one processor, the electronic device: The operation of determining a vector representing the speed of the seek mode using the ratio of the amount of change between the continuously acquired seek times (Δs) and the amount of change between the continuously acquired request times (Δr); and An operation to determine the Decision Intensity, which represents the magnitude of the mobility of the seek mode, and the Decision Direction, which represents the direction of movement of the seek mode, using the vector determined above. An electronic device that causes to perform.

4. In Paragraph 3, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation to update a vector representing the velocity of the seek mode, a Decision Intensity representing the magnitude of the mobility of the seek mode, and a Decision Direction representing the direction of movement of the seek mode whenever a seek request is received; and An operation to initialize a vector representing the velocity of the seek mode, a Decision Intensity representing the magnitude of the mobility of the seek mode, and a Decision Direction representing the direction of movement of the seek mode in response to the last seek request. An electronic device that causes to perform.

5. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Operation to check the interval for the period of an I-frame (I-frame Interval); An operation to check the average decoding time of the above electronic device; and An operation to determine the seek mode by comparing the above-determined interval and the above-determined average decoding time, An electronic device that causes to perform.

6. In Paragraph 5, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation to determine the seek mode by comparing the above-mentioned confirmed interval and the above-mentioned confirmed average decoding time with a vector representing the speed of the seek mode, a Decision Intensity representing the magnitude of the mobility of the seek mode, and a Decision Direction representing the direction of movement of the seek mode according to a predetermined standard. An electronic device that causes to perform.

7. In Paragraph 6, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation to turn on the optimization mode in response to the above intensity being greater than the above interval; and The operation of determining the seek mode as an I-frame seek for the above optimization mode, An electronic device that causes to perform.

8. In Paragraph 6, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation to set the optimization mode off in response to the above intensity being smaller than the above interval; and The operation of determining the seek mode as a closed seek to disable the above optimization mode, An electronic device that causes to perform.

9. In Paragraph 6, When the above instructions are executed individually or collectively by at least one processor, the electronic device: If the above vector is greater than a predetermined reference value, the operation of determining that the movement direction of the above seek mode has a positive direction and determining the above seek mode as the next sync seek; and If the above vector is smaller than the above predetermined reference value, the operation of determining that the movement direction of the above seek mode has a negative direction and determining the above seek mode as a previous sync seek, An electronic device that causes to perform.

10. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Action of determining a rendering optimization mode based on rendering time and decoding frame extraction time for decoded frames and rendered frames, An electronic device that causes to perform.

11. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Based on the above-determined seek mode, the operation of determining the optimization mode by adjusting the range of the optimization level for rendering and the rendering optimization level according to the characteristics of each video edit, An electronic device that causes to perform.

12. In Paragraph 11, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation to check the rendering delay level by considering the number of decoded frames; An operation to adjust the optimization level for rendering in consideration of the above-mentioned confirmed delay level; and The operation of determining the optimization mode by adjusting the range of rendering optimization levels according to the characteristics of each video edit, An electronic device that causes to perform.

13. In Paragraph 12, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation to variably set the characteristics of each video edit according to the performance of the video editor and the quality of service corresponding to user requirements; and An operation of selectively displaying a video with applied rendering editing characteristics by considering the optimization level adjusted for each of the above video editing characteristics and the range of the optimization level, Causing to perform, An electronic device wherein each of the above-mentioned video editing characteristics includes at least one of pen characteristics, stroke type, and alpha blending.

14. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device, An operation of applying an optimized seek operation according to the determined seek mode and the optimization mode; and An action of displaying a video with an optimized rendering operation applied by applying an optimization level and a range of optimization levels adjusted according to the characteristics of each video edit, An electronic device that causes to perform.

15. In a method of displaying an image, An operation to acquire Seek Time and Request Real Time; An operation to determine a seek mode for applying an optimization mode by considering the ratio of the time for the above-mentioned obtained seek time and the above-mentioned obtained request real time; and An operation to output a screen based on a requested frame based on time information for the frame of the above-determined seek mode, A display method including

Images