Electronic device supporting selective video post-processing and operating method thereof

Abstract

The present disclosure provides a method performed by an electronic device. The method may comprise the operations of: obtaining a video frame by decoding a compressed video frame of a video; identifying whether the video frame corresponds to the last frame of the video being repeatedly played back; and on the basis of identifying that the video frame corresponds to the last frame, processing such that no interpolation frame is generated between the last frame and a start frame, which is the next frame of the last frame.

Description

Electronic device and method of operation supporting selective video post-processing

[0001] The present disclosure relates to an electronic device and a method of operation that supports selective video post-processing.

[0002] Recently, the demand for high-quality video playback has significantly increased due to advancements in display technology and the widespread availability of high-definition and ultra-high-definition content. To meet this demand, video post-processing technologies such as noise reduction, upscaling, color enhancement, and frame rate conversion are being developed. These technologies can primarily be applied to enhance the viewing experience on display devices, including televisions, monitors, and mobile devices. However, due to their high dependence on hardware, these post-processing technologies are generally difficult to apply flexibly and adaptively to specific situations.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] According to one embodiment of the present disclosure, an electronic device comprises at least one processor including a processing circuit; and a memory including at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to perform at least one operation. The at least one operation may include an operation of decoding a compressed video frame of a video to obtain a video frame. The at least one operation may include an operation of identifying whether the video frame corresponds to the last frame of the video being played repeatedly. The at least one operation may include an operation of processing so that an interpolated frame is not generated between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to the last frame.

[0005] According to one embodiment, a method of operating an electronic device may be provided. The method of operating an electronic device may include at least one operation. The at least one operation may include an operation of decoding a compressed video frame of a video to obtain a video frame. The at least one operation may include an operation of identifying whether the video frame corresponds to the last frame of the video being played repeatedly. The at least one operation may include an operation of processing so that an interpolated frame is not generated between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to the last frame.

[0006] According to one embodiment, a storage medium may be provided for storing at least one instruction readable by a computer. The at least one instruction may cause the electronic device to perform at least one operation when executed by at least a part of at least one processor of the electronic device. The at least one operation may include an operation of decoding a compressed video frame of a video to obtain a video frame. The at least one operation may include an operation of identifying whether the video frame corresponds to the last frame of the video being played repeatedly. The at least one operation may include an operation of processing so that an interpolated frame is not generated between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to the last frame.

[0007] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0009] FIG. 2 is a drawing illustrating a video stream according to one embodiment of the present disclosure.

[0010] FIG. 3 is a drawing illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart illustrating the operation of an electronic device performing selective post-processing according to one embodiment of the present disclosure.

[0012] FIG. 5 is a flowchart illustrating the operation of an electronic device to selectively generate an interpolation frame according to one embodiment of the present disclosure.

[0013] FIG. 6a is a flowchart illustrating an operation to determine whether an electronic device will generate an interpolation frame according to one embodiment of the present disclosure.

[0014] FIG. 6b is a flowchart illustrating an operation to determine whether an electronic device will generate an interpolation frame, according to one embodiment of the present disclosure.

[0015] FIG. 7 is a signal flow diagram illustrating a procedure in which an electronic device generates an interpolated frame using an FRC device according to one embodiment of the present disclosure.

[0016] FIG. 8 is a signal flow diagram illustrating a procedure for an electronic device to allocate a buffer according to one embodiment of the present disclosure.

[0017] FIG. 9 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0018] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0019] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0020] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0021] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0022] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0023] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0024] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0025] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0026] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0027] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0028] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0029] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0030] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0031] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0032] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0033] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0034] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0035] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0036] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a communication module (192) (e.g., cellular communication module, short-range communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0037] The communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0038] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0039] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0040] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface) and exchange signals (e.g., commands or data) with each other.

[0041] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0042] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0043] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0044] FIG. 2 is a drawing illustrating a video stream according to one embodiment of the present disclosure.

[0045] According to one embodiment, the video stream may be a video stream for a video that is played repeatedly. For example, the video stream may be a video stream for a short video that is played repeatedly. The video stream may include a plurality of video frames (e.g., 101 video frames), for example, as illustrated in part (a) of FIG. 2. As illustrated, the timestamp value of the first video frame (first frame) may be 0ms, and the timestamp value of the last video frame (last frame) may be 3300ms. The total length (duration) of the video stream may be, for example, 3333ms. In the present disclosure, a video frame may be abbreviated as a frame.

[0046] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may call a codec flush to shift the position of the video when replaying the video, after playing up to the last frame of the video stream and restarting from the first frame (start frame). When decoding is restarted from the first frame by calling a codec flush in this way, a delay in video output may occur. For example, depending on the performance of the electronic device, a delay of generally 50 ms to 80 ms may occur. This may give the user the impression that the video is playing in a choppy manner. To solve this problem, when replaying the video, instead of calling a codec flush, a method of changing only the timestamp value of the first frame (e.g., changing from 0 ms to 3333 ms) may be used, as exemplified in part (b) of FIG. 2. This method may be used, for example, when the first frame is an IDR (instantaneous decoder refresh), but is not limited thereto. The IDR frame can be independently decoded and acts as the starting point of the group of pictures (GOP) without relying on the previous GOP, and can be used to start a new session or jump to a specific location during video playback. Referring to part (b) of Fig. 2, after playback from the first frame (#1) to the last frame (#101), the timestamp value of the first frame (#1) can be changed from 0ms to 3333ms without a codec flush. This allows the user to be provided with a seamless video output experience even when moving from the last frame to the first frame and playing again.

[0047] Meanwhile, interpolated frames can be generated for the entire looped video. For example, interpolated frames can be generated for a looped video using FRC technology. However, generating interpolated frames for a looped video may lead to certain issues. For instance, during the generation process, an unnecessary interpolated frame may be created between the last frame and the first frame. Additionally, if the images at the beginning and end are similar, an incorrect interpolated frame may be generated. Therefore, to resolve these issues, it is necessary to consider a method to selectively generate interpolated frames for specific sections or specific frames, rather than for the entire video.

[0048] As another example of an issue, video quality may deteriorate due to the application of video post-processing (e.g., upscaling) in situations where changes occur during video playback, such as scale adjustments (e.g., zoom-in), resolution changes, playback speed changes, or FPS changes. To resolve this issue, it is necessary to consider a method to selectively apply video post-processing to specific sections or frames as needed, in order to provide users with a seamless video output experience.

[0049] In the following description, for convenience of explanation, FRC technology (e.g., FRC up-conversion technology) is used as an example to describe a technology for generating interpolated frames, but is not limited thereto. Various types of technologies for generating interpolated frames may be applied to various embodiments of the present disclosure. In the following description, for convenience of explanation, video and video frames are used as examples of applicable images and image frames; however, various embodiments of the present disclosure may be applied to various types of images, such as still images or video, and corresponding image frames, provided that they are not contradictory.

[0050] FIG. 3 is a drawing illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0051] Referring to FIG. 3, according to one embodiment, an electronic device (101) may include a data source processor (310), a demuxer (320), a decoder (330), a parser (340), a post-processor (350), a renderer (360), a buffer allocator (370), and / or a frame rate conversion (FRC) device (380). In the present disclosure, the demuxer (320) may be referred to as a de-multiplexer.

[0052] According to one embodiment, a data source processor (310) can obtain a bit stream (or, data stream) for video playback from data received from a video providing device (e.g., server (108) of FIG. 1) via a network. The data source processor (310) can obtain a bit stream (or, video stream) for video playback from a video file stored in an electronic device (101). The data source processor (310) can store the bit stream in a buffer to pass to a demuxer (320).

[0053] According to one embodiment, the bitstream may include at least one data stream and metadata. The at least one data stream may include a video stream, an audio stream, and / or a subtitle stream. The video stream may include encoded video data (e.g., compressed video frames). The audio stream may include encoded audio data (e.g., compressed audio frames). The subtitle stream may include text and / or image-based subtitle data. The metadata may include information associated with the corresponding data stream. For example, metadata associated with the video stream may include timing information, codec information (e.g., H.264, H.265), and / or video attribute information (e.g., resolution, frame rate).

[0054] According to one embodiment, the demuxer (320) may request the data source processor (310) to obtain a bit stream and separate metadata and each data stream from the bit stream. For example, the demuxer (320) may separate from the bit stream a video stream containing compressed video frame(s) and metadata related to the video stream. The video stream may be passed to the decoder (330), and the related metadata may be passed to the parser (340). For convenience of explanation, the description of the processing of the audio stream and subtitle stream among the data streams separated from the demuxer (320) is omitted below, and the subsequent processing of the video stream is described as an example.

[0055] According to one embodiment, the demuxer (320) (or electronic device (101)) checks for changes in network speed and, if necessary, can selectively request low-quality video data or high-quality video data from the video provider. For example, the demuxer (320) can request high-quality video data from the video provider if the network speed is maintained at a higher than reference speed for a specified period of time. For example, the demuxer (320) can request low-quality video data from the video provider if the network speed is maintained at a lower than reference speed for a specified period of time.

[0056] According to one embodiment, the decoder (330) can decode the compressed video frame(s). For example, the decoder (330) can decode the compressed video frame(s) using codec information. Through this, the original video frame(s) can be restored.

[0057] According to one embodiment, the parser (340) can parse metadata and provide information contained in the metadata to the decoder (330), postprocessor (350), and / or renderer (360). For example, the parser (340) can transmit codec information contained in the metadata to the decoder (330). For example, the parser (340) can transmit video attribute information contained in the metadata to the postprocessor (350).

[0058] According to one embodiment, the post-processor (350) may perform post-processing on the video frame(s). For example, the post-processor (350) may perform (or apply) selective post-processing on the video frame(s). Post-processing may be performed to improve the quality of the decoded video frames or to adjust them so that they are properly output through a display. Post-processing may include, but is not limited to, resolution adjustment, interpolation frame generation, frame rate conversion (FRC), color space conversion, and / or filtering and enhancement. Hereinafter, an interpolation frame generation operation related to FRC is described as an example of a post-processing operation.

[0059] According to one embodiment, the post-processor (350) may include a meta information generator (351), a post-processing controller (352), and / or an FRC controller (353).

[0060] According to one embodiment, a meta information generator (351) may acquire (e.g., generate) meta information used for post-processing of a video frame. The meta information may include information set in units of individual video frames (e.g., timestamp information) and / or information set in units of multiple video frames (e.g., total duration information). The meta information generator (351) may acquire or generate meta information associated with each video frame on a per-video frame basis. The meta information generator (351) may generate meta information using metadata, video frames, information regarding the playback status of the video (e.g., current playback time, buffer status, playback status, screen mode), and / or related information. The related information may include at least one piece of information transmitted to the meta information generator (351) from at least one other configuration. At least one other configuration may include, but is not limited to, an application, a display manager, and / or a device thermal manager. In the present disclosure, the meta information used for post-processing of a video frame may be referred to as post-processing meta information. According to one embodiment, the meta information generator (351) can essentially generate or acquire meta information for a video frame when an application (e.g., the application (146) of FIG. 1), a demuxer (320) and / or a decoder (330) is used.

[0061] According to one embodiment, the meta information generator (351) can store meta information in memory (e.g., memory (130) of FIG. 1). According to one embodiment, the meta information generator (351) can share (e.g., transmit to another electronic device (102)) the meta information with another electronic device (e.g., electronic device (102) of FIG. 1). For example, in a scenario (e.g., mirroring scenario) where only a decoding operation for a video frame is performed by the electronic device (101) and subsequent operations (e.g., post-processing, rendering) are performed by another electronic device (102), the electronic device (101) can transmit the meta information along with the video frame to the other electronic device (102). In this case, the other electronic device (102) can selectively perform post-processing on the video frame based on the received meta information, and then perform rendering to output the video frame through a display.

[0062] According to one embodiment, meta information for a video frame (or meta information associated with a video frame) may include, for example, information set for an individual video frame (e.g., timestamp information) and / or information set for a plurality of video frames including the video frame (e.g., total duration information). The meta information may include resolution information for the video frame, view size information within the display, FPS (frames per second) information for the video frame, total duration information of the video, information on whether the video is repeated, temperature information for the electronic device (101), initial timestamp information of the video, timestamp information for the video frame, playback speed information for the video frame, maximum display refresh rate information, color format information for the video frame, and / or information on whether post-processing is enabled (e.g., flag information). The meta information may be used by a post-processing controller (352) to determine whether to apply post-processing to the video frame. Through this meta information, post-processing for each video frame may be selectively performed.

[0063] According to one embodiment, the resolution information may include information indicating the resolution of a video frame (e.g., 1920 x 1080), or information regarding the width (e.g., 1920) and height (e.g., 1080) of a video frame. According to one embodiment, the meta information generator (351) may generate or obtain resolution information based on metadata associated with a video stream.

[0064] According to one embodiment, view size information within a display may indicate the view size at which a video frame is output. The view size may be defined by a view width and a view height. The view width and the view height may be set, for example, within the maximum configurable width and maximum configurable height of the display, respectively. For example, if the maximum configurable width and the maximum configurable height of the display are 1920 and 1080 pixels, respectively, the view width and the view height may be set within 1920 and 1080 pixels, respectively (e.g., set to 1600 and 1080 pixels). According to one embodiment, the renderer (360) may output a video frame with the size changed to the view size based on the view size information within the display. In the present disclosure, the view size information within the display may be abbreviated as display view size information or view size information.

[0065] According to one embodiment, FPS information may indicate the FPS of a video frame (e.g., 30 fps, 60 fps). According to one embodiment, a meta information generator (351) may generate or obtain FPS information based on metadata associated with a video stream or information received from an application.

[0066] According to one embodiment, the total duration information may indicate the total duration (length) (or total playback duration) (e.g., 513666 microseconds) of a video containing video frames. According to one embodiment, the meta information generator (351) may generate or obtain the total duration information based on metadata associated with the video stream (e.g., metadata included in the header of the video stream).

[0067] According to one embodiment, repeat playback information may indicate whether the video is repeated. For example, the repeat playback information may be set to a first value (e.g., 0) indicating that the video is not repeated, or a second value (e.g., 1) indicating that the video is repeated. According to one embodiment, a meta information generator (351) may generate or obtain repeat playback information based on information received from an application providing the video (e.g., a video application).

[0068] According to one embodiment, the initial time stamp information of the video may include information indicating the initial time stamp of the video. The initial time stamp of the video may, for example, correspond to the time stamp of the first frame of the video being played when the video is played. The initial time stamp may, for example, be a specified value greater than or equal to 0 (e.g., 0ms or 10000000 ms). Generally, the initial time stamp of the video being played may be 0ms, but since it may be intentionally set to a value greater than 0ms (e.g., 10000000 ms) if necessary, it is necessary to acquire or generate the initial time stamp information of the video indicating the set value as meta information. According to one embodiment, the meta information generator (351) may generate or acquire the initial time stamp information based on metadata associated with the video stream or information received from an application providing the video (e.g., a video application).

[0069] According to one embodiment, temperature information may indicate the temperature of the electronic device (101). According to one embodiment, a meta-information generator (351) may generate or obtain temperature information based on information received from a device temperature manager.

[0070] According to one embodiment, the time stamp information may indicate the time stamp of a video frame (e.g., 33333 microseconds). According to one embodiment, the meta information generator (351) may generate or obtain the time stamp information of individual video frames based on information received from a video stream (e.g., metadata included in the frame header of each video frame within the video stream) or a demuxer (320).

[0071] According to one embodiment, playback speed information may indicate the playback speed of a video frame (e.g., 0.25x, 0.5x, 1x, 2x). According to one embodiment, a meta information generator (351) may generate or obtain playback speed information based on metadata associated with a video stream, information received from a display manager, or an application.

[0072] According to one embodiment, the maximum display refresh rate information may indicate a maximum display refresh rate (e.g., 60 Hz, 120 Hz). According to one embodiment, the meta information generator (351) may generate or obtain the maximum display refresh rate information based on metadata associated with the video stream or information received from the display manager.

[0073] According to one embodiment, color format information may indicate the color format of a video frame (e.g., yuv, yv12, nv12). According to one embodiment, a meta information generator (351) may generate or obtain color format information based on information received from metadata associated with a video stream.

[0074] According to one embodiment, information on whether post-processing is enabled may include information that explicitly or directly indicates whether to enable post-processing for a video frame. For example, information on whether post-processing is enabled may be set to a first value (e.g., 0) indicating that post-processing for a video frame is not to be performed, or a second value (e.g., 1) indicating that post-processing for a video frame is to be performed. According to one embodiment, a meta-information generator (351) may generate or obtain information on whether post-processing is enabled based on information received from an application inside or outside the electronic device (101) (e.g., an explicit request from the application). For example, the meta-information generator (351) may receive information on whether post-processing is enabled generated from an application inside or outside the electronic device (101) (e.g., a video application) and include the received information on whether post-processing is enabled in the meta-information.

[0075] According to one embodiment, the post-processing controller (352) can control post-processing for a video frame. The post-processing controller (352) can selectively perform post-processing for a video frame. The post-processing controller (352) can perform at least one of the following processing operations to selectively perform post-processing for a video frame. The at least one processing operation may include, for example, an operation of acquiring a video frame, an operation of acquiring meta-information for a video frame, an operation of determining whether to perform (or apply) post-processing for the video frame based on the meta-information, and / or an operation of performing or not performing post-processing on the video frame based on the determination. According to one embodiment, the operation of determining whether to perform post-processing for the video frame may include an operation of determining whether the video frame corresponds to the last frame or the start frame using the video total duration information of the video. Through this operation, post-processing for each video frame may be selectively applied.

[0076] According to one embodiment, the FRC controller (353) can control processing (post-processing) related to FRC. The FRC controller (353) can manage an FRC device (380) used for processing related to FRC (e.g., interpolation frame generation processing). When the FRC device (380) is ready, the FRC controller (353) can transmit a notification message to the post-processing controller (352) indicating that the FRC device (380) is in a ready state (e.g., a state where interpolation frames can be generated). The FRC controller (353) can selectively generate at least one interpolation frame associated with a video frame using the FRC device (380). For example, the FRC controller (353) can control the FRC device (380) to selectively generate at least one interpolation frame associated with a video frame based on a control signal from the post-processing controller (352). The FRC controller (353) may use a bypass flag transmitted from the post-processing controller (352) for the selective generation of interpolated frames. According to one embodiment, the post-processing controller (352) may transmit a control signal containing a bypass flag, or a control signal containing a bypass flag set to a first value (e.g., 1), to the FRC controller (352) if it does not want to generate an interpolated frame for the corresponding video frame while the FRC device (380) is in a ready state. In this case, the FRC controller (352) may identify the bypass flag and, based on the bypass flag, omit or bypass the operation to generate an interpolated frame for the corresponding video frame using the FRC device (380).According to one embodiment, if the post-processing controller (352) wants to generate an interpolated frame for the corresponding video frame while the FRC device (380) is in a ready state, it may transmit to the FRC controller (352) a control signal that does not include a bypass flag or a control signal that includes a bypass flag set to a second value (e.g., 0). In this case, the FRC controller (352) may identify that the bypass flag is not included or is set to a second value, and based on the identification result, perform an operation to generate an interpolated frame for the corresponding video frame using the FRC device (380). The FRC controller (353) may provide information about the capability (or spec) that the FRC controller (353) can support. Information regarding the performance that the FRC controller (353) can support may include information regarding the supported resolution (e.g., width and / or height) and / or information regarding the supported interpolation frame generation scale (e.g., x2 (2x), x4 (4x)). If the information regarding the interpolation frame generation scale is set to 2x, one interpolation frame may be generated between two original frames. For example, through the 2x setting, it may consist of frames such as original frame 1, interpolation frame 1-1, original frame 2, interpolation frame 2-1, ... If the information regarding the interpolation frame generation scale setting is set to 4x, three interpolation frames may be generated between two original frames. For example, through a 4x setting, it can be composed of frames such as original frame 1, interpolated frame 1-1, interpolated frame 1-2, interpolated frame 1-3, original frame 2, interpolated frame 2-1, interpolated frame 2-2, interpolated frame 2-3, ... In the present disclosure, the interpolated frame generation scale may correspond to an up factor.

[0077] According to one embodiment, the renderer (360) can render a video frame for output (e.g., display) through a display (e.g., the display module (160) of FIG. 1). The renderer (360) can render a video frame that has been post-processed through a post-processor (350). For example, the renderer (360) can render video frame(s) and interpolated frame(s) generated through the post-processor (350).

[0078] According to one embodiment, the buffer allocator (370) may provide a buffer for decoding, post-processing, and / or rendering. An example of the buffer allocation operation of the buffer allocator (370) is described below with reference to FIG. 8.

[0079] According to one embodiment, the FRC device (380) may be a device used to convert the frame rate of a video. The FRC device (380) may convert the frame rate of an input video (e.g., 30 fps) to the frame rate of an output video (e.g., 60 fps) based on a control signal of the FRC controller (353). The control signal may include, for example, a bypass flag for the corresponding video frame. The conversion of the frame rate may include, for example, an up conversion that generates at least one interpolated frame associated with an input frame, or a down conversion that deletes at least one of the input frames. The FRC device (370) may convert the frame rate of the video using techniques such as frame duplication, frame drop, and motion interpolation.

[0080] According to one embodiment, each of the above-described components may be implemented by at least one processor (e.g., processor (120) of FIG. 1) and / or memory (e.g., memory (130) of FIG. 1). For example, a data source processor (310), a demuxer (320), a decoder (330), a parser (340), a postprocessor (350), a renderer (360), a buffer allocator (370), and / or an FRC device (380) may be implemented by a single processor (e.g., an AP). For example, the meta-information generator (351), the post-processing controller (352), and the renderer (360) of the postprocessor (350) may be implemented by the AP, and the FRC controller (353) and the FRC device (380) of the postprocessor (350) may be implemented by an NPU, a DSP, and / or a GPU. For example, the buffer allocator (370) can be implemented by memory.

[0081] FIG. 4 is a flowchart illustrating the operation of an electronic device performing selective post-processing according to one embodiment of the present disclosure.

[0082] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0083] Referring to FIG. 4, according to one embodiment, in operation 410, an electronic device (e.g., the electronic device (101) of FIG. 1) may perform a decoding operation. The decoding operation may include, for example, an operation of decoding a compressed video frame to obtain a video frame. According to one embodiment, operation 410 may be performed by the decoder (330) of FIG. 3.

[0084] According to one embodiment, in operation 420, the electronic device (101) may perform an optional post-processing operation. The optional post-processing operation may include, for example, an operation of acquiring a video frame, an operation of acquiring meta-information for the video frame, an operation of determining whether to perform (or apply) post-processing to the video frame based on the meta-information, and / or an operation of performing or not performing post-processing on the video frame based on the determination. According to one embodiment, the operation of determining whether to perform post-processing on the video frame may include an operation of determining whether the video frame corresponds to the last frame or the start frame using the video total duration information of the video. The post-processing may include, for example, resolution adjustment, interpolation frame generation, frame rate conversion, color space conversion, and / or filtering and image quality improvement, but is not limited thereto. According to one embodiment, the electronic device (101) may selectively perform (or apply) post-processing on a per-video frame basis according to the situation, taking into account changes in the situation such as changes in video resolution, changes in playback speed, and / or changes in FPS during video playback, based on meta-information. For a description of the meta-information, refer to the description in FIG. 3. According to one embodiment, operation 420 may be performed by the post-processor (350) of FIG. 3 or the post-processing controller (352) of the post-processor (350).

[0085] According to one embodiment, in operation 430, the electronic device (101) may perform a rendering operation. The rendering operation may include, for example, an operation of rendering a video frame on which an optional post-processing operation has been performed. According to one embodiment, operation 430 may be performed by the renderer (360) of FIG. 3. According to one embodiment, the electronic device (101) may store data of the post-processed video frame. For example, the data may include a video frame and at least one interpolated frame generated based on the video frame when post-processing to generate an interpolated frame for the video frame is applied. For example, the data may include a video frame when post-processing to generate an interpolated frame for the video frame is not applied.

[0086] The operations 410 to 430 of FIG. 4 described above can be performed for each video frame. For example, the electronic device (101) can perform a decoding operation, a selective post-processing operation, and a rendering operation for a first video frame to output the first video frame through a display, and perform a decoding operation, a selective post-processing operation, and a rendering operation for a second video frame, which is a frame after the first video frame, to output the second video frame through a display. Through the selective post-processing operation of operation 420, which is performed between the decoding operation of operation 410 and the rendering operation of operation 430, each video frame included in the video stream can be selectively post-processed. For example, some video frames of the video can be post-processed, while other video frames may not be post-processed. In this way, instead of the entire video being post-processed or not post-processed, only specific frames or specific sections of the video can be selectively post-processed according to specific situations.

[0087] In the following, the operation of an electronic device performing selective post-processing is described exemplarily, using the processing for generating interpolated frames as an example of post-processing.

[0088] FIG. 5 is a flowchart illustrating the operation of an electronic device to selectively generate an interpolation frame according to one embodiment of the present disclosure.

[0089] The embodiment of FIG. 5 may be, for example, an example of the selective post-processing operation of FIG. 4. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0090] Referring to FIG. 5, according to one embodiment, in operation 510, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a video frame by decoding a compressed video frame. Operation 510 may be, for example, an example of operation 410 of FIG. 4. According to one embodiment, operation 510 may include all or part of the operations performed by the decoder (330) of FIG. 3.

[0091] According to one embodiment, in operation 520, the electronic device (101) may obtain meta-information for a video frame. According to one embodiment, a meta-information generator (e.g., the meta-information generator (351) of FIG. 3) may generate meta-information for a video frame based on (or by reference) metadata, information about the video frame, the playback status of the video, and / or related information. For a description of the meta-information, refer to the description of FIG. 3. Operation 520 may be, for example, an example of operation 420 of FIG. 4. According to one embodiment, operation 520 may include all or part of the operations performed by the post-processor (350) of FIG. 3 or the meta-information generator (351) of the post-processor (350).

[0092] According to one embodiment, in operation 530, the electronic device (101) may selectively generate at least one interpolated frame associated with a video frame based on meta-information. According to one embodiment, the operation of selectively generating at least one interpolated frame associated with a video frame may include an operation of determining whether to generate at least one interpolated frame for the video frame. An explanation of the operation is given below with reference to FIG. 6. Operation 530 may be, for example, an example of operation 420 of FIG. 4. According to one embodiment, operation 530 may include all or part of the operations performed by the post-processor (350) of FIG. 3 or the post-processing controller (352) of the post-processor (350).

[0093] According to one embodiment, the operation of selectively generating at least one interpolated frame may include: an operation of identifying, based on meta-information, that a video associated with a video frame is a video that is being played repeatedly; an operation of identifying, based on meta-information, whether a video frame corresponds to a designated frame (e.g., last frame or start frame) of a video that is being played repeatedly; an operation of processing such that at least one interpolated frame is not generated between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to a designated frame (e.g., last frame or start frame); and an operation of processing such that at least one interpolated frame associated with a frame is generated based on the identification that the video frame does not correspond to a designated frame (e.g., last frame and / or start frame). The operation of processing so that at least one interpolation frame is not generated may include, for example, an operation in which the post-processing controller (352) transmits a control signal to the FRC controller (353) that includes a bypass flag or a bypass flag set to a first value (e.g., 1) instructing that at least one interpolation frame is not generated through the FRC device (380) (e.g., instructing to bypass the generation of the interpolation frame), or an operation in which the post-processing controller (352) omits transmitting the control signal itself to the FRC controller (353). The operation of processing so that at least one interpolation frame is not generated may include, for example, an operation in which the post-processing controller (352) transmits a request to the FRC controller (353) to stop the generation of the interpolation frame when the interpolation frame is being generated. The operation of processing to generate at least one interpolated frame may include, for example, the operation of the post-processing controller (352) transferring a buffer for the video frame and a buffer for generating the interpolated frame to the FRC controller (353) (e.g., operations 812 and 813 of FIG. 8).The operation of processing to generate at least one interpolation frame may include, for example, the operation of the post-processing controller (352) transmitting a control signal that does not include a bypass flag to the FRC controller (353), or transmitting a control signal that includes a bypass flag set to a second value (e.g., 0).

[0094] According to one embodiment, the meta-information may include total duration information indicating the total duration of the video and / or timestamp information indicating the timestamp of the video frame. An operation for identifying whether a video frame corresponds to the last frame or the start frame may include an operation for identifying whether a video frame corresponds to the last frame or the start frame based on the total duration information and / or timestamp information. An example of an operation for identifying whether a video frame corresponds to the last frame or the start frame based on the total duration information and / or timestamp information is described exemplarily below with reference to Equation 1.

[0095] According to one embodiment, the meta information may include repeat playback status information indicating whether the video is being repeated. The operation of selectively generating at least one interpolated frame may include an operation of identifying that the video is a video being repeated based on the repeat playback status information.

[0096] According to one embodiment, meta-information includes flag information (post-processing activation information) indicating whether to generate at least one interpolation frame associated with a video frame, and the flag information may be set to a first value indicating to generate at least one interpolation frame associated with a video frame or a second value indicating not to generate at least one interpolation frame associated with a video frame. An operation to selectively generate at least one interpolation frame may include an operation to process so that at least one interpolation frame is not generated based on the flag information being set to the first value, and an operation to process so that at least one interpolation frame is generated based on the flag information being set to the second value.

[0097] According to one embodiment, the generation of at least one interpolated frame is performed using a frame rate conversion technique, and the meta-information includes up factor information indicating an up factor of the frame rate conversion technique, and the number of at least one interpolated frame can be set based on the up factor information.

[0098] According to one embodiment, in operation 540, the electronic device (101) may include an operation of rendering a video frame. According to one embodiment, operation 530 may include all or part of the operations performed by the renderer (360) of FIG. 3. According to one embodiment, the electronic device (101) may store data of a post-processed video frame. For example, the data may include a video frame and at least one interpolated frame generated based on the video frame, where post-processing to generate an interpolated frame for the video frame is applied. For example, the data may include a video frame, where post-processing to generate an interpolated frame for the video frame is not applied.

[0099] The operations 510 to 540 of FIG. 5 described above can be performed for each video frame. For example, the electronic device (101) can output the first video frame through a display by performing a decoding operation, a meta-information acquisition operation, an optional interpolation frame generation operation, and a rendering operation for the first video frame, and can output the second video frame through a display by performing a decoding operation, a meta-information acquisition operation, an optional interpolation frame generation operation, and a rendering operation for the second video frame, which is a frame after the first video frame. Through the meta-information generation operation and the optional interpolation frame generation operation of operations 520 and 530, which are performed between the decoding operation of operation 510 and the rendering operation of operation 540, at least one interpolation frame associated with each video frame included in the video stream can be selectively generated. For example, at least one interpolation frame associated with some video frames of the video can be generated, and at least one interpolation frame associated with other video frames may not be generated. In this way, instead of a method in which interpolated frames are generated for the entire video or not generated, interpolated frames for specific frames or specific sections of the video may be selectively generated to suit specific situations. Meanwhile, for example, in the case of the mirroring scenario described above, operations 510 and 520 may be performed by the electronic device (101), and operations 530 and 540 may be performed by another electronic device (e.g., the electronic device (102) of FIG. 1) connected to the electronic device (101). In this case, the electronic device (101) may transmit the acquired meta-information along with the video frames to the other electronic device (102).

[0100] FIG. 6a is a flowchart illustrating an operation to determine whether an electronic device will generate an interpolation frame according to one embodiment of the present disclosure.

[0101] The embodiment of FIG. 6a may be an example of operation 530 of FIG. 5. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0102] Referring to FIG. 6a, in operation 610a, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a video frame and meta-information about the video frame. The electronic device (101) can obtain the video frame and meta-information, for example, through operations 510 and 520 of FIG. 5.

[0103] According to one embodiment, in operation 620a, the electronic device (101) may determine whether to generate an interpolated frame for a video frame based on meta-information. If it is determined that an interpolated frame for a video frame is generated, operation 630a may be performed. If it is determined that an interpolated frame for a video frame is not generated, operation 640a may be performed. Operation 620a may be, for example, an example of operation 530 of FIG. 5. According to one embodiment, operation 620a may include all or part of the operations performed by the post-processor (350) of FIG. 3 or the post-processing controller (352) of the post-processor (350).

[0104] According to one embodiment, the operation of determining whether to generate an interpolated frame for a video frame based on the meta information of operation 620a may include at least one of the following determination operations. The at least one determination operation may include, for example, a first determination operation based on FPS information, a second determination operation based on resolution information, a third determination operation based on playback speed information, a fourth determination operation based on temperature information, a fifth determination operation based on display refresh rate, a sixth determination operation based on view size information within the display, a seventh determination operation related to FRC device preparation, or at least one of an eighth determination operation related to the last frame or the start frame of a video being played repeatedly.

[0105] According to one embodiment, the above-described at least one decision operation may be composed of various numbers, sequences, and combinations. For example, only some of the plurality of decision operations may be performed to determine whether to generate an interpolation frame. For example, the first through fourth decision operations, the sixth decision operation, the seventh decision operation, and the eighth decision operation, excluding the fifth decision operation, may be performed sequentially to determine whether to generate an interpolation frame. For example, the plurality of decision operations may be performed simultaneously or in parallel with each other. For example, at least two of the first through sixth decision operations may be performed in parallel with each other. For example, some decision operations may be performed prior to other decision operations. For example, the seventh decision operation related to FRC device preparation may be performed prior to at least one of the other decision operations (e.g., the eighth decision operation). For example, the first through sixth decision operations may be performed prior to at least one of the other decision operations (e.g., the seventh decision operation and / or the eighth decision operation).

[0106] According to one embodiment, the electronic device (101) may determine to generate an interpolated frame for a video frame when at least one of at least one judgment condition corresponding to at least one judgment operation is satisfied together or simultaneously. For example, the first to fourth judgment operations, the sixth judgment operation, the seventh judgment operation, and the eighth judgment operation, excluding the fifth judgment operation, may be performed sequentially to determine whether to generate an interpolated frame, and when the first to fourth judgment conditions, the sixth judgment condition, the seventh judgment condition, and the seventh judgment condition corresponding to the first to fourth judgment operations, the sixth judgment operation, the seventh judgment operation, and the eighth judgment operation are all satisfied, it may determine to generate an interpolated frame for a video frame. For example, the electronic device (101) may perform only the seventh and eighth judgment operations among the first to eighth judgment operations to determine whether to generate an interpolated frame for a video frame, and may determine to generate an interpolated frame for a video frame when the seventh judgment condition corresponding to the seventh judgment operation and the eighth judgment condition corresponding to the eighth judgment operation are satisfied together.

[0107] According to one embodiment, the electronic device (101) may decide not to generate an interpolated frame for a video frame if any one of at least one judgment condition corresponding to at least one judgment operation is not satisfied. For example, the first to fourth judgment operations, the sixth judgment operation, the seventh judgment operation, and the eighth judgment operation, excluding the fifth judgment operation, may be performed sequentially to determine whether to generate an interpolated frame, and if any one of the first to fourth judgment conditions, the sixth judgment condition, the seventh judgment condition, and the eighth judgment condition corresponding to the first to fourth judgment operations, the sixth judgment operation, the seventh judgment operation, and the eighth judgment operation is not satisfied, it may decide not to generate an interpolated frame for a video frame. For example, the electronic device (101) may perform only the seventh and eighth judgment operations among the first to eighth judgment operations to determine whether to generate an interpolated frame for a video frame, and if either the seventh judgment condition corresponding to the seventh judgment operation or the eighth judgment condition corresponding to the eighth judgment operation is not satisfied, it may decide not to generate an interpolated frame for a video frame.

[0108] The descriptions of each judgment action and the corresponding judgment conditions are explained by example below.

[0109] According to one embodiment, the electronic device (101) may obtain performance information (e.g., spec) related to the support of interpolation frame generation (hereinafter, interpolation frame generation support information) before performing the at least one judgment operation described above or together with performing the at least one judgment operation. The electronic device (101) may obtain interpolation frame generation support information from an FRC controller (e.g., the FRC controller (353) of FIG. 3). The interpolation frame generation support information may include, for example, a minimum width (e.g., 640 pixels), a minimum height (e.g., 480 pixels), a maximum width (3840 pixels), a maximum height (2160 pixels), an FPS (e.g., 30 or 60 fps), and / or a maximum up factor (e.g., 4) that is supported for interpolation frame generation. As described above, when the up factor is 4, three interpolation frames may be generated between two consecutive original frames.

[0110] According to one embodiment, the electronic device (101) can determine whether a judgment condition corresponding to a judgment operation is satisfied based on interpolation frame generation support information and meta information. According to one embodiment, the electronic device (101) can determine whether a judgment condition corresponding to a judgment operation is satisfied by comparing interpolation frame generation support information and meta information. Below, each judgment operation is described by example.

[0111] According to one embodiment, in a first determination operation, the electronic device (101) can compare the FPS value of the FPS information in the meta information (e.g., 30 fps) with the FPS value of the interpolation frame generation support information (e.g., 30 fps) to determine whether a first determination condition corresponding to the first determination operation (e.g., FRC applicable FPS condition) is satisfied. For example, if the FPS value of the FPS information in the meta information (e.g., 30 fps) belongs to the FPS value of the interpolation frame generation support information (e.g., 30 fps), that is, if the FPS value is supported, the electronic device (101) can determine that the first determination condition corresponding to the first determination operation is satisfied. If the FPS value of the FPS information in the meta information (e.g., 120 fps) does not belong to the FPS value of the interpolation frame generation support information (e.g., 30 fps), that is, if the FPS value is not supported, the electronic device (101) may determine that the first judgment condition corresponding to the first judgment operation is not satisfied.

[0112] According to one embodiment, in a second determination operation, the electronic device (101) can compare the resolution value of the resolution information in the meta-information (e.g., 1080 x 1920) with the resolution value of the interpolation frame generation support information to determine whether a second determination condition corresponding to the second determination operation (e.g., a resolution condition applicable to FRC) is satisfied. If the resolution value of the resolution information in the meta-information (e.g., 1080 x 1920) falls within the range of the resolution value of the interpolation frame generation support information, that is, if the corresponding resolution value is supported, the electronic device (101) can determine that the second determination condition corresponding to the second determination operation is satisfied. If the resolution value of the resolution information in the meta-information (e.g., 1080 x 1920) does not fall within the range of the resolution value of the interpolation frame generation support information, that is, if the corresponding resolution value is not supported, the electronic device (101) can determine that the second determination condition corresponding to the second determination operation is not satisfied.

[0113] According to one embodiment, in a third determination operation, the electronic device (101) can determine whether a third determination condition corresponding to the third determination operation (e.g., an FRC applicable playback speed condition) is satisfied by using the playback speed value of the playback speed information in the meta information and the FPS value of the FPS information. For example, if the playback speed value of the playback speed information in the meta information is 0.25x and the FPS value of the FPS information is 30 fps, the electronic device (101) can determine that the third determination condition corresponding to the third determination operation is satisfied because it is possible to support generating an interpolated frame using an up-factor 4, which is within the range of up-factors that can be supported by playing the video at 1 / 4 speed. Through this, playback of a video at 30 fps at 0.25x speed may be possible. For example, if the playback speed value of the playback speed information in the meta information is 0.5x and the FPS value of the FPS information is 30 fps, the electronic device (101) can be supported in generating interpolated frames using an up-factor 2 that is within the range of supported up-factors, so that the video is played at 1 / 2 speed, and thus the third judgment condition corresponding to the third judgment operation can be determined to be satisfied. Through this, playback of a video at 30 fps at 0.25 speed can be made possible. For example, if the playback speed value of the playback speed information in the meta information is 1x and the FPS value of the FPS information is 30 fps, the electronic device (101) can be supported in generating interpolated frames using an up-factor 2 that is within the range of supported up-factors for smoother output, so that the third judgment condition corresponding to the third judgment operation can be determined to be satisfied. Through this, playback of a video at 60 fps at 1 speed can be made possible.

[0114] According to one embodiment, in a fourth determination operation, the electronic device (101) can compare the temperature value of the temperature information in the meta information with the maximum allowable device temperature value (e.g., 40 degrees) to determine whether a fourth determination condition (e.g., FRC applicable temperature condition) corresponding to the fourth determination operation is satisfied. If the temperature value of the temperature information in the meta information (e.g., 37 degrees) is lower than the maximum allowable device temperature value (e.g., 40 degrees), the electronic device (101) can determine that the fourth determination condition corresponding to the fourth determination operation is satisfied. It can determine to generate an interpolated frame for the video frame. If the temperature value of the temperature information in the meta information (e.g., 42 degrees) is higher than the maximum allowable device temperature value (e.g., 40 degrees), the electronic device (101) can determine that the fourth determination condition corresponding to the fourth determination operation is not satisfied.

[0115] According to one embodiment, in the fifth determination operation, the electronic device (101) can determine whether a fifth determination condition corresponding to the fifth determination operation (e.g., an FRC-applicable display refresh rate condition) is satisfied by using the display refresh rate value of the display refresh rate information in the meta information and the FPS value of the FPS information. For example, if the display refresh rate value of the display refresh rate information in the meta information is 60 Hz and the FPS value of the FPS information is 30 fps, the electronic device (101) can determine that only 60 video frames per second are output by the display refresh rate when changing the FPS value to 120 fps by applying an up factor 4 that is within the range of a supportable up factor. In this case, the electronic device (101) can determine that the fifth determination condition corresponding to the fifth determination operation is not satisfied. In other words, the electronic device (101) can determine that interpolated frames are not generated to exceed the display refresh rate.

[0116] According to one embodiment, in the sixth determination operation, the electronic device (101) can compare the view size value of the view size information in the meta information with the size value of the entire display to determine whether a sixth determination condition corresponding to the sixth determination operation (e.g., a view size condition for FRC application) is satisfied. For example, if the view size value of the view size information in the meta information exceeds a specified ratio (e.g., 10%) of the size value of the entire display, it can be determined that the sixth determination condition corresponding to the sixth determination operation is satisfied. For example, if the view size value of the view size information in the meta information (e.g., 192 x 108) is less than or equal to a specified ratio (e.g., 10%) of the size value of the entire display (e.g., 1920 x 1080), it can be determined that the sixth determination condition corresponding to the sixth determination operation is not satisfied. Through this, if the size of the display view output in the display is less than or equal to a certain ratio (e.g., 10%) of the total display size, the size of the output screen is too small, so FRC may not be applied unnecessarily.

[0117] According to one embodiment, in the seventh judgment operation, the electronic device (101) may determine whether a seventh judgment condition corresponding to the seventh judgment operation (e.g., FRC device ready condition) is satisfied based on whether the FRC device (e.g., FRC device (380) of FIG. 3) is ready. If the FRC device is ready, the electronic device (101) may determine that the seventh judgment condition corresponding to the seventh judgment operation is satisfied. If the FRC device is not ready, the electronic device (101) may determine that the seventh judgment condition corresponding to the seventh judgment operation is not satisfied.

[0118] According to one embodiment, in the eighth judgment operation, the electronic device (101) may determine whether an eighth judgment condition corresponding to the eighth judgment operation (e.g., a condition satisfying the last frame or start frame of the video being played repeatedly) is satisfied based on whether the video frame corresponds to the last frame or start frame of the video being played repeatedly. If the video frame corresponds to the last frame or start frame of the video being played repeatedly, the electronic device (101) may determine that the eighth judgment condition corresponding to the eighth judgment operation is satisfied. If the video frame does not correspond to the last frame or start frame of the video being played repeatedly, the electronic device (101) may determine that the eighth judgment condition corresponding to the eighth judgment operation is not satisfied.

[0119] According to one embodiment, the electronic device (101) can determine whether a video frame is the last frame or the starting frame of a video that is being played repeatedly by using the following mathematical formula 1.

[0120]

[0121] Here,

[0122] video total duration: Value of the total duration information of the video within the metadata

[0123] video timestamp: Value of the timestamp information for a video frame within the metadata (corresponds to the timestamp value of the current video frame)

[0124] base timestamp: The timestamp value of the first video frame (start video frame) where loop playback begins.

[0125] video previous timestamp: timestamp value of the previous video frame

[0126] According to one embodiment, the electronic device (101) can determine that the current video frame is the last frame or the starting frame of the video being repeated if "video total duration - adjustedTime < video timestamp - video previous timestamp" or "adjustedTime >= video total duration". Otherwise, the electronic device (101) can determine that the current video frame is not the last frame or the starting frame of the video being repeated. For example, if the value of the total duration information (Video Total Duration) is 3330 ms, the value of the current video timestamp is 3300 ms, the video previous timestamp is 3267 ms, and the base timestamp is 0 ms, then video total duration - adjustedTime (= 30 ms = 3330 ms - 3300 ms) < video timestamp - video previous timestamp (= 33 ms = 3300 ms - 3267 ms), so it can be determined that the corresponding video frame is the last frame. For example, if the value of the total duration information (Video Total Duration) is 3290 ms, the value of the current video timestamp is 3300 ms, and the base timestamp is 0 ms, then since adjustedTime(= 3300 = 3300-0) >= video total duration(=3290ms), it can be determined that the video frame corresponds to the last frame.

[0127] According to one embodiment, the electronic device (101) may set the timestamp of the start frame, which is the next frame of the last frame, as the base timestamp. For example, as illustrated in part (b) of FIG. 2, the timestamp of the start frame, which is the next frame of the last frame, may be set to 3330 ms as the base timestamp. According to one embodiment, in operation 630a, the electronic device (101) may perform an interpolation frame generation operation. For example, the electronic device (101) may generate at least one interpolation frame associated with a video frame (e.g., a frame other than the last frame and start frame of a looped video). The operation of generating at least one interpolation frame may include, for example, an operation in which a post-processing controller (352) passes a buffer for the video frame and / or a buffer for generating the interpolation frame to an FRC controller (353) (e.g., operations 812 and 813 of FIG. 8). The operation of processing to generate at least one interpolation frame may include, for example, the operation of the post-processing controller (352) transmitting a control signal that does not include a bypass flag to the FRC controller (353), or transmitting a control signal that includes a bypass flag set to a second value (e.g., 0).

[0128] According to one embodiment, in operation 640a, the electronic device (101) may not perform an interpolation frame generation operation. For example, the electronic device (101) may bypass the operation of generating at least one interpolation frame associated with a video frame (e.g., the last frame or the start frame of a looped video). The operation of not generating at least one interpolation frame may include, for example, an operation in which a post-processing controller (352) transmits a control signal including a bypass flag or a bypass flag set to a first value (e.g., 1) to the FRC controller (353) instructing that at least one interpolation frame not be generated through the FRC device (380) (e.g., instructing to bypass the generation of the interpolation frame), or an operation in which the post-processing controller (352) omits transmitting the control signal itself to the FRC controller (353). The operation to prevent at least one interpolated frame from being generated may include, for example, an operation in which, if an interpolated frame is being generated, the post-processing controller (352) transmits a request to the FRC controller (353) to stop the generation of the interpolated frame. Through this, an interpolated frame may not be generated between the last frame and the start frame of the video being played repeatedly.

[0129] The operations 610a to 640a of FIG. 6a described above can be performed for each video frame. Through this, at least one interpolated frame associated with each video frame included in the video stream can be selectively generated.

[0130] FIG. 6b is a flowchart illustrating an operation to determine whether an electronic device will generate an interpolation frame, according to one embodiment of the present disclosure.

[0131] The embodiment of FIG. 6b may be an example of operation 530 of FIG. 5. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0132] Referring to FIG. 6b, in operation 610b, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a video frame and meta-information about the video frame. The electronic device (101) can obtain the video frame and meta-information, for example, through operations 510 and 520 of FIG. 5.

[0133] According to one embodiment, in operation 620b, the electronic device (101) may determine whether to generate an interpolated frame for a video frame based on meta-information. If it is determined that an interpolated frame for a video frame is generated, operation 630b may be performed. If it is determined that an interpolated frame for a video frame is not generated, operation 640b may be performed. Operation 620b may be, for example, an example of operation 530 of FIG. 5. According to one embodiment, operation 620b may include all or part of the operations performed by the post-processor (350) of FIG. 3 or the post-processing controller (352) of the post-processor (350).

[0134] According to one embodiment, the operation of determining whether to generate an interpolated frame for a video frame based on the meta-information of operation 620b may include at least one of the following determination operations. The at least one determination operation may include, for example, a first determination operation based on FPS information, a second determination operation based on resolution information, a third determination operation based on playback speed information, a fourth determination operation based on temperature information, a fifth determination operation based on the display refresh rate, or a sixth determination operation based on view size information within the display. For a description of each determination operation and the corresponding determination condition of the first to sixth determination operations, refer to the description in FIG. 6a. A description redundant thereto is omitted.

[0135] According to one embodiment, the above-described at least one judgment operation may be composed of various numbers, sequences, and combinations. For example, only some of the plurality of judgment operations may be performed to determine whether to generate an interpolation frame. For example, the first to sixth judgment operations may be performed sequentially to determine whether to generate an interpolation frame. For example, the plurality of judgment operations may be performed simultaneously or in parallel. For example, at least two of the first to sixth judgment operations may be performed in parallel. For example, some judgment operations may be performed prior to other judgment operations.

[0136] According to one embodiment, the electronic device (101) may determine to generate an interpolated frame for a video frame when at least one of at least one judgment condition corresponding to at least one judgment operation is satisfied together or simultaneously. For example, the first to sixth judgment operations may be performed sequentially to determine whether to generate an interpolated frame, and when all of the first to sixth judgment conditions corresponding to the first to sixth judgment operations are satisfied, it may be determined to generate an interpolated frame for a video frame.

[0137] According to one embodiment, the electronic device (101) may decide not to generate an interpolated frame for a video frame if any one of at least one judgment condition corresponding to at least one judgment operation is not satisfied. For example, the first to sixth judgment operations may be performed sequentially to determine whether to generate an interpolated frame, and if any one of the first to sixth judgment conditions corresponding to the first to sixth judgment operations is not satisfied, it may be decided not to generate an interpolated frame for a video frame.

[0138] According to one embodiment, in operation 630b, the electronic device (101) can determine whether the FRC device (e.g., the FRC device (380) of FIG. 3) is ready. According to one embodiment, the electronic device (101) can determine that the FRC device (380) is ready if the post-processing controller (e.g., the post-processing controller (352) of FIG. 3) receives an FRC device ready response (FRC ready) from the FRC controller (e.g., the FRC controller (353) of FIG. 3). If the FRC device (380) is ready, operation 650b may be performed. If the FRC device (380) is not ready, operation 640b may be performed.

[0139] According to one embodiment, in operation 640b, the electronic device (101) may not perform an interpolation frame generation operation. According to one embodiment, operations 620b and 640b may be performed before the FRC device (380) is ready. For example, operations 620b and 640b may be performed before the post-processing controller (352) transmits an FRC device initialization request (FRC init) to the FRC controller (353), or after the post-processing controller (352) transmits an FRC device initialization request (FRC init) to the FRC controller (353) but before receiving an FRC device ready response (FRC ready). In this case, in operation 640b, the electronic device (101) may not transmit a control signal for interpolation frame generation to the FRC controller (353) because the FRC device (380) is not ready. As a result, the interpolation frame generation operation through the FRC controller (353) may not be performed.

[0140] According to one embodiment, in operation 650b, the electronic device (101) can determine whether the video frame is the last frame or the start frame. For a description of operation 650b, refer to the description of operation 530 of FIG. 5 and operation 620 of FIG. 6a. If the video frame is the last frame or the start frame, operation 670b may be performed. If the video frame is not the last frame and the start frame, operation 660b may be performed.

[0141] According to one embodiment, in operation 660b, the electronic device (101) may perform an interpolation frame generation operation based on the identification that the video frame is not the last frame and the start frame. For example, the electronic device (101) may generate at least one interpolation frame associated with the video frame (e.g., a frame that is not the last frame and the start frame of a looped video). The operation of generating at least one interpolation frame may include, for example, an operation in which a post-processing controller (352) passes a buffer for the video frame and / or a buffer for generating the interpolation frame to an FRC controller (353) (e.g., operations 812 and 813 of FIG. 8). The operation of processing to generate at least one interpolation frame may include, for example, an operation in which the post-processing controller (352) passes a control signal that does not include a bypass flag to the FRC controller (353), or a control signal that includes a bypass flag set to a second value (e.g., 0).

[0142] According to one embodiment, in operation 670b, the electronic device (101) may not generate (e.g., bypass) a specific interpolation frame based on the identification of the last frame or the start frame of the video frame. For example, the electronic device (101) may bypass the operation of generating at least one interpolation frame associated with the video frame (e.g., the last frame or the start frame of a looped video) when the FRC device (380) is ready. The operation of bypassing the generation of at least one interpolation frame may include, for example, the operation of a post-processing controller (352) transmitting a control signal to the FRC controller (353) including a bypass flag or a bypass flag set to a first value (e.g., 1) that instructs the FRC device (380) not to generate at least one interpolation frame (e.g., to bypass the generation of the interpolation frame). The operation of bypassing the generation of at least one interpolated frame may include, for example, an operation in which, if an interpolated frame is being generated, the post-processing controller (352) transmits a request to the FRC controller (353) to stop the generation of the interpolated frame. Through this, an interpolated frame may not be generated between the last frame and the start frame of the video being played repeatedly.

[0143] FIG. 7 is a signal flow diagram illustrating a procedure in which an electronic device generates an interpolated frame using an FRC device according to one embodiment of the present disclosure.

[0144] In the embodiment of FIG. 7, for convenience of explanation, the up factor for generating interpolated frames is described as being 2, but is not limited thereto. Even if a different up factor is set within the range of up factors (e.g., 4) that can be supported by an FRC device (e.g., the FRC device (380) of FIG. 3), the description of the embodiment of FIG. 7 may be applied substantially the same way.

[0145] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0146] Referring to FIG. 7, in operation 701, the decoder (330) can transmit a first video frame (video frame #1), which is the first video frame (start video frame) of the video stream, to the post-processing controller (352). In operation 702, the post-processing controller (352) can transmit an FRC device initialization request (FRC init) to the FRC controller (353) by calling a command to initialize the FRC device. The FRC controller (353) can perform an operation to initialize the FRC device based on the FRC device initialization request.

[0147] According to one embodiment, the FRC device may be implemented as a processor such as an NPU or GPU. Therefore, it may take a certain amount of time (e.g., hundreds of milliseconds) for the FRC device to be ready for use. Consequently, if an interpolated frame for a video frame is generated after waiting for the FRC device to be ready, the video may stop for about hundreds of milliseconds to 1 second. Accordingly, to ensure uninterrupted playback of the video, the post-processing controller (352) may not perform the operation of generating an interpolated frame until an FRC device ready response (FRC ready) is received from the FRC controller (353) in response to the FRC device initialization request (FRC init), indicating that the FRC device is ready. For example, the post-processing controller (350) may not perform an interpolation frame generation operation for the second video frame (video frame #2) to the fifth video frame (video frame #5) received via operations 703, 705, 707, and 709 before receiving an FRC device ready response (FRC ready) via operation 711, and may transmit the second decoding result (decoding result #2) to the fifth decoding result (decoding result #5) corresponding to the second video frame to the fifth video frame via operations 704, 706, 708, and 710 to the renderer (360). According to one embodiment, the second decoding result to the fifth decoding result may each include the second video frame to the fifth video frame. The renderer (360) may sequentially render and output the received video frames. Through this, the video can be output without interruption.

[0148] According to one embodiment, in operation 712, the decoder (330) can transmit the sixth video frame (video frame #6) to the post-processing controller (352). Since the sixth video frame is a frame received after the FRC device is prepared, the post-processing controller (352) can perform an interpolation frame generation operation for the sixth video frame through the FRC controller (353) (e.g., the optional interpolation frame generation operation of FIGS. 5 and 6). In operation 713, the post-processing controller (352) can transmit the sixth video frame to the FRC controller (353). The post-processing controller (352) can transmit meta-information for the sixth video frame along with the sixth video frame to the FRC controller (353). In operation 714, the post-processing controller (352) may send a request to the FRC controller (353) to allocate a buffer (buffer #6-1) for an interpolation frame associated with the sixth video frame. In operation 715, the post-processing controller (352) may send a seventh video frame (video frame #7), which is a frame after the sixth video frame, to the FRC controller (353). The post-processing controller (352) may send meta information for the seventh video frame along with the seventh video frame to the FRC controller (353). In operation 716, the post-processing controller (352) may send a request to the FRC controller (353) to allocate a buffer (buffer #7-1) for an interpolation frame associated with the seventh video frame.

[0149] According to one embodiment, the FRC controller (353) can use an FRC device to generate an interpolated frame (interpolated frame #6-1) associated with the sixth video frame and generate an interpolated frame (interpolated frame #7-1) associated with the seventh video frame. In operation 717, the FRC controller (353) can transmit the sixth video frame to the post-processing controller (352). In operation 718, the post-processing controller (352) can transmit the sixth video frame to the renderer (360). In operation 719, the FRC controller (353) can transmit the interpolated frame associated with the sixth video frame to the post-processing controller (352). In operation 720, the post-processing controller (352) can transmit the interpolated frame associated with the sixth video frame to the renderer (360). In operation 721, the FRC controller (353) can transmit the seventh video frame to the post-processing controller (352). In operation 722, the post-processing controller (352) can transmit the seventh video frame to the renderer (360). In operation 723, the FRC controller (353) can transmit the interpolated frame associated with the seventh video frame to the post-processing controller (352). In operation 724, the post-processing controller (352) can transmit the interpolated frame associated with the seventh video frame to the renderer (360). Through this, the video frame and the associated interpolated frame can be sequentially rendered and output by the renderer (360).

[0150] FIG. 8 is a signal flow diagram illustrating a procedure for an electronic device to allocate a buffer according to one embodiment of the present disclosure.

[0151] According to one embodiment, the decoder (330) may receive a buffer (e.g., a surface buffer) through the buffer allocator (370) to minimize memory copying during video playback, and fill the allocated buffer with the decoding result and request rendering from the renderer (360).

[0152] According to one embodiment, the decoder (330) and the renderer (360) may specify a certain number of buffers for decoding and rendering, allocate buffers within the specified number of buffers, and perform the operation of reusing the buffers. In this buffer structure, a post-processor (350) may be additionally used for post-processing, such as generating optional interpolated frames. The post-processor (350) may, for example, request the allocation of buffers for interpolated frames. In this case, the specified number of buffers may all be preempted by the decoder (330) and / or the renderer (360), so that the post-processor (350) may not be allocated buffers for interpolated frames. Therefore, in order for the post-processor (350) to easily secure buffers for interpolated frames, a method such as the procedure of FIG. 8 needs to be considered.

[0153] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0154] In the embodiment of FIG. 8, for convenience of explanation, the up factor for generating interpolated frames is described as being 2, but is not limited thereto. Even if a different up factor is set within the range of up factors (e.g., 4) that can be supported by an FRC device (e.g., the FRC device (380) of FIG. 3), the description of the embodiment of FIG. 8 may be applied substantially the same way.

[0155] Referring to FIG. 8, according to one embodiment, in operation 801, the decoder (330) may transmit information (hereinafter referred to as the first buffer number information) regarding the number of buffers required for decoding (e.g., 20) to the buffer allocator (370). In operation 802, the post-processing controller (352) may transmit information (hereinafter referred to as the second buffer number information) regarding the number of buffers required for post-processing (e.g., post-processing for generating selective interpolated frames using FRC) (e.g., 10) to the buffer allocator (370). In operation 803, the buffer allocator (370) may set the total number of buffers (e.g., 30) based on the first buffer number information and the second buffer number information, and set the number of buffers required (or required for post-processing) (e.g., 10) by the post-processing controller (352). The buffer setting information set through this buffer setting process may be as shown in Table 1 below.

[0156] MaxBuffer: 30MaxFRCBuffer: 10DequeueableBuffer: 30DequeuedBuffer: 0FrcReservedBuffer: 0

[0157] Here, the maximum number of buffers named "MaxBuffer" may correspond to the total number of buffers set by the buffer allocator (370). The maximum number of FRC buffers named "MaxFRCBuffer" may correspond to the number of buffers required for post-processing (e.g., FRC) set by the buffer allocator (370). The number of queuable buffers named "DequeueableBuffer" may correspond to the number of currently queuable buffers set within the maximum number of buffers. The number of queued buffers named "DequeuedBuffer" may correspond to the number of currently queued buffers set within the maximum number of buffers. The number of FRC reserved buffers named "FrcReservedBuffer" may correspond to the number of currently reserved buffers for post-processing (e.g., FRC) set within the maximum number of FRC buffers.

[0158] According to one embodiment, in operation 804, the decoder (330) may request a buffer for decoding (e.g., an output buffer) from the buffer allocator (370) at the start of decoding. In operation 805, the buffer allocator (370) may allocate a buffer for decoding.

[0159] According to one embodiment, in operation 806, the buffer allocator (380) may reserve a buffer for an interpolation frame. For example, if the up factor for an interpolation frame is 2, the buffer allocator (380) may reserve one buffer for one interpolation frame, as illustrated in FIG. 8. The buffer setting information changed through such buffer allocation and reservation may be as shown in Table 2 below.

[0160] MaxBuffer: 30MaxFRCBuffer: 10DequeueableBuffer: 28DequeuedBuffer: 1FrcReservedBuffer: 1

[0161] Unlike as exemplified in FIG. 8, the up factor for interpolation frames may be greater than 2. In this case, the buffer allocator (380) may reserve a number of buffers corresponding to the number of up factors. For example, if the up factor for interpolation frames is 4, the buffer allocator (380) may reserve 3 buffers for 3 interpolation frames. In this case, unlike in Table 2, the number of FRC reserved buffers (FrcReservedBuffer) may be set to 3, and the number of queuable buffers (DequeueableBuffer) may be set to 26. Depending on the embodiment, operation 806 may be omitted. For example, if the maximum number of FRC buffers (MaxFRCBuffer) and the number of FRC reserved buffers (FrcReservedBuffer) are the same (e.g., 10), the buffer allocator (380) may not reserve buffers for interpolation frames, as all buffers for interpolation frames are already reserved.

[0162] According to one embodiment, in operation 807, the buffer allocator (370) may transfer the allocated output buffer to the decoder (330). In operation 808, the decoder (330) completes decoding the compressed first video frame (video frame #1) using the allocated output buffer and may transfer the decoded first video frame to the post-processing controller (352). In operation 809, the post-processing controller (352) may request a buffer from the buffer allocator (370) for an interpolated frame for the first video frame (e.g., the sixth frame or the seventh frame of FIG. 7). Operation 809 may be, for example, an example of operation 714 or operation 716 of FIG. 7. In operation 810, the buffer allocator (370) may allocate the buffer for the interpolated frame reserved through operation 806 as a buffer for generating an interpolated frame for the first video frame. In this case, the buffer allocator (370) may set the number of FRC reserved buffers (FrcReservedBuffer) to 0 and the number of queued buffers (DequeuedBuffer) to 2. In operation 811, the buffer allocator (370) may transfer the buffer allocated for the interpolated frame for the first video frame to the post-processing controller (352). The buffer setting information changed through this buffer allocation process may be as shown in Table 3 below.

[0163] MaxBuffer: 30MaxFRCBuffer: 10DequeueableBuffer: 28DequeuedBuffer: 2FrcReservedBuffer: 0

[0164] Through the operations described above, the post-processing controller (352) can be guaranteed the allocation of a buffer for the interpolation frame. If these operations are summarized in a formula, they may be as shown in Tables 4 and 5 below.

[0165] ·Request buffer from decoderupFactor (number of fps up, eg 2 or 4)if (FrcReservedBuffer < MaxFrcBuffer) {if (DequeueableBuffer <= upFactor - 1) { / original return BLOCKING / buffer allocation impossible}FrcReservedBuffer++DequeueableBuffer--Surface Buffer allocation (dequeueBuffer)DequeueableBuffer--}

[0166] ·Request buffer from post processing controllerIf (FrcReservedBuffer > 0) {Surface Buffer allocation (dequeueBuffer)FrcReservedBuffer--}

[0167] Table 4 illustrates a formula for buffer allocation related to a buffer request from the decoder (330) (e.g., a buffer request for decoding). Referring to Table 4, "If the number of FRC reserved buffers (FrcReservedBuffer) is less than the maximum number of FRC buffers (MaxFRCBuffer):

[0168] If the number of queuable buffers (DequeueableBuffer) is less than the upFactor minus 1, return BLOCKING (buffer cannot be allocated), and

[0169] Otherwise, increase the number of FRC reserved buffers (FrcReservedBuffer), decrease the number of queuable buffers (DequeueableBuffer), allocate a new buffer (e.g., surface buffer), and decrease the number of queuable buffers (DequeueableBuffer).

[0170] Table 5 illustrates a formula for buffer allocation related to a buffer request from the post-processing controller (352) (e.g., a buffer request for FRC). Referring to Table 5, "if the number of FRC reserved buffers (FrcReservedBuffer) is greater than 0, allocate a new buffer (e.g., a surface buffer) and decrease the number of FRC reserved buffers (FrcReservedBuffer)."

[0171] According to one embodiment, in operation 812, the post-processing controller (352) may transmit the first video frame to the FRC controller (353). According to one embodiment, if the post-processing controller (352) decides to generate an interpolated frame for the first video frame based on meta-information for the first video frame, the post-processing controller (352) may transmit the first video frame to the FRC controller (353) for generating the interpolated frame. In operation 813, the post-processing controller (352) may transmit a buffer allocated for the interpolated frame to the FRC controller (353). The FRC controller (353) may generate an interpolated frame for the first video frame through the FRC device (380) using the first video frame and the buffer allocated for the interpolated frame. Meanwhile, as described above, if the post-processing controller (352) decides not to generate an interpolation frame for the first video frame based on the meta-information for the first video frame, operations 812 and 813 may not be performed. Instead, the post-processing controller (352) may transmit a bypass flag to the FRC controller (353) instructing that at least one interpolation frame not be generated through the FRC device. If an interpolation frame is being generated, the post-processing controller (352) may transmit a request for the post-processing controller to stop generating the interpolation frame to the FRC controller (353).

[0172] FIG. 9 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0173] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0174] Referring to FIG. 9, in operation 910, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a video frame by decoding a compressed video frame of the video. The electronic device (101) can obtain a video frame, for example, through operation 410 of FIG. 4 or operation 510 of FIG. 5.

[0175] In operation 920, the electronic device (101) can identify whether a video frame corresponds to the last frame of a video that is being played repeatedly. According to one embodiment, operation 920 may include an operation of obtaining meta-information about a video frame and an operation of identifying whether the video frame corresponds to the last frame of a video that is being played repeatedly based on the meta-information. The electronic device (101) can obtain meta-information about a video frame, for example, through operation 520 of FIG. 5.

[0176] According to one embodiment, meta information for a video frame (or meta information associated with a video frame) may include, for example, information set for an individual video frame (e.g., timestamp information) and / or information set for a plurality of video frames including said video frame (e.g., total duration information).

[0177] According to one embodiment, the meta information may include total duration information indicating the total duration of the video and timestamp information indicating the timestamp of the video frame. An operation to identify whether it corresponds to the last frame may include an operation to identify whether the video frame corresponds to the last frame based on the total duration information and the timestamp information.

[0178] According to one embodiment, the meta information may include repeat playback status information indicating whether the video is being repeated. The electronic device (101) can identify that the video is a video that is being repeated based on the repeat playback status information.

[0179] According to one embodiment, the electronic device (101) can reset the value of the timestamp of the start frame based on the identification that the video frame corresponds to the last frame.

[0180] According to one embodiment, meta information may include flag information indicating whether to generate at least one interpolation frame associated with a video frame. The flag information may be set to a first value indicating to generate at least one interpolation frame associated with a video frame or a second value indicating not to generate at least one interpolation frame associated with a video frame. The electronic device (101) may process so that at least one interpolation frame is not generated based on the flag information being set to the first value, and may process so that at least one interpolation frame is generated based on the flag information being set to the second value.

[0181] According to one embodiment, the generation of at least one interpolated frame may be performed using a frame rate conversion technique. Meta information may include up factor information indicating an up factor of the frame rate conversion technique. The number of at least one interpolated frame may be set based on the up factor information. According to one embodiment, when allocating a buffer for decoding a video frame, the electronic device (101) may reserve at least one buffer for the generation of at least one interpolated frame.

[0182] In operation 930, the electronic device (101) can process so that no interpolated frame is generated between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to the last frame.

[0183] In operation 940, the electronic device (101) can process the video frame so that an interpolated frame is created between the video frame and the subsequent frame based on the identification that the video frame does not correspond to the last frame.

[0184] According to one embodiment of the present disclosure, an electronic device comprises at least one processor including a processing circuit; and a memory including at least one storage medium for storing instructions, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause: to decode a compressed video frame to obtain a video frame, to obtain meta information for said video frame, to selectively generate at least one interpolation frame associated with said video frame based on said meta information, and to render said video frame.

[0185] According to one embodiment, meta information for a video frame (or meta information associated with a video frame) may include, for example, information set for an individual video frame (e.g., timestamp information) and / or information set for a plurality of video frames including said video frame (e.g., total duration information).

[0186] According to one embodiment, the operation of selectively generating at least one interpolation frame may include: an operation of identifying, based on the meta-information, whether the video frame corresponds to the last frame of a video that is being repeated; and an operation of processing so that at least one interpolation frame is not generated between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to the last frame. The operation of processing so that at least one interpolation frame is not generated may include, for example, an operation in which a post-processing controller transmits a bypass flag to an FRC controller instructing that at least one interpolation frame is not generated through an FRC device. The operation of processing so that at least one interpolation frame is not generated may include, for example, an operation in which, when an interpolation frame is being generated, the post-processing controller transmits a request to the FRC controller (353) to stop the generation of the interpolation frame.

[0187] According to one embodiment, the operation of selectively generating the at least one interpolated frame may include an operation of processing to generate the at least one interpolated frame based on the identification that the video frame does not correspond to the last frame. The operation of processing to generate the at least one interpolated frame may include, for example, an operation in which a post-processing controller transmits a buffer for the video frame and a buffer for generating the interpolated frame to an FRC controller.

[0188] According to one embodiment, the meta information may include total duration information indicating the total duration of the video and timestamp information indicating the timestamp of the video frame. The operation of identifying whether it corresponds to the last frame may include an operation of identifying whether the video frame corresponds to the last frame based on the total duration information and timestamp information.

[0189] According to one embodiment, the meta information may include repeat playback status information indicating whether the video is played repeatedly. The operation of selectively generating at least one interpolated frame may include an operation of identifying that the video is a video that is played repeatedly based on the repeat playback status information.

[0190] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause: based on the identification that the video frame corresponds to the last frame, the value of the timestamp of the start frame to be reset.

[0191] According to one embodiment, the meta information includes flag information indicating whether to generate at least one interpolation frame associated with the video frame, and the flag information may be set to a first value indicating to generate at least one interpolation frame associated with the video frame or a second value indicating not to generate at least one interpolation frame associated with the video frame.

[0192] According to one embodiment, the operation of selectively generating the at least one interpolated frame may include: an operation of processing so that the at least one interpolated frame is not generated based on the flag information being set to the first value and the video frame being identified as corresponding to the last frame; and an operation of processing so that the at least one interpolated frame is generated based on the flag information being set to the second value and the video frame being identified as not corresponding to the last frame.

[0193] According to one embodiment, the generation of at least one interpolated frame is performed using a frame rate conversion technique, and the meta-information includes up factor information indicating an up factor of the frame rate conversion technique, and the number of at least one interpolated frame can be set based on the up factor information.

[0194] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: reserve at least one buffer for generating the at least one interpolated frame when allocating a buffer for decoding the video frame.

[0195] According to one embodiment, the meta information can be generated per video frame.

[0196] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0197] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0198] One embodiment of the present document may be implemented as software (e.g., program (140) of FIG. 1) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) of FIG. 1 or external memory (138) of FIG. 1) that is readable by a machine (e.g., electronic device (101) of FIG. 1). For example, a processor (e.g., processor (120) of FIG. 1) of the machine (e.g., electronic device (101) of FIG. 1) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0199] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0200] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, At least one processor including a processing circuit; and The electronic device comprises a memory including at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, cause: Decoding the compressed video frames of the video to obtain video frames, and Identify whether the above video frame corresponds to the last frame of the above video that is being played repeatedly, and An electronic device that causes processing to prevent the creation of an interpolation frame between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to the last frame.

2. In Paragraph 1, The operation of identifying whether the above video frame corresponds to the last frame of the above video is, An operation to acquire meta information associated with the above video frame; and An electronic device comprising an operation to identify whether the video frame corresponds to the last frame of a video being repeatedly played, based on the above meta information.

3. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the video frame to be processed such that an interpolated frame is generated between the video frame and a subsequent frame, based on the identification that the video frame does not correspond to the last frame.

4. In Paragraph 2 or 3, The above meta information includes total duration information indicating the total duration of the video and timestamp information indicating the timestamp of the video frame, and An electronic device, wherein the operation of identifying whether the above video frame corresponds to the last frame includes the operation of identifying whether the above video frame corresponds to the last frame based on the above total period information and the above timestamp information.

5. In Paragraph 4, The above meta information includes repeat playback status information indicating whether the video is played repeatedly, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes to identify that the video is a video that is being played repeatedly based on the information regarding whether the video is being played repeatedly.

6. In any one of paragraphs 2 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the value of the timestamp of the start frame to be reset based on the identification that the video frame corresponds to the last frame.

7. In Paragraph 1, The above meta information includes flag information indicating whether to generate at least one interpolation frame associated with the video frame, and the flag information is set to a first value indicating to generate at least one interpolation frame associated with the video frame or a second value indicating not to generate at least one interpolation frame associated with the video frame. When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Based on the fact that the above flag information is set to the above first value, processing is performed so that the above at least one interpolation frame is not generated, and An electronic device that causes processing to generate at least one interpolation frame based on the above flag information being set to the above second value.

8. In any one of paragraphs 1 through 7, The generation of at least one interpolated frame is performed using frame rate conversion technology, and The above meta-information includes up factor information indicating an up factor of the frame rate conversion technology, and The number of at least one interpolation frame is set based on the upward factor information, in an electronic device.

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes at least one buffer to be reserved for the generation of at least one interpolated frame when allocating a buffer for decoding the above video frame.

10. In any one of paragraphs 1 through 9, The above meta information is generated per video frame by an electronic device.

11. In a method of an electronic device, The operation of obtaining a video frame by decoding a compressed video frame of a video; An operation to identify whether the above video frame corresponds to the last frame of the above video that is repeatedly played; and A method comprising an operation to prevent an interpolation frame from being generated between the last frame and the start frame, which is the next frame of the last frame, based on the identification that the video frame corresponds to the last frame.

12. In Paragraph 11, The operation of identifying whether the above video frame corresponds to the last frame of the above video is, An operation to acquire meta information associated with the above video frame; and A method comprising an operation to identify whether the video frame corresponds to the last frame of a video that is repeatedly played, based on the above meta information.

13. In paragraph 11, the above method is: A method comprising processing the video frame such that at least one interpolated frame is generated between the video frame and a subsequent frame based on the identification that the video frame does not correspond to the last frame.

14. In Paragraph 12 or 13, The above meta information includes total duration information indicating the total duration of the video and timestamp information indicating the timestamp of the video frame, and A method for identifying whether the video frame corresponds to the last frame, wherein the operation of identifying whether the video frame corresponds to the last frame is based on the entire period information and the timestamp information.

15. In Paragraph 14, The above meta information includes repeat playback status information indicating whether the video is repeated, and the method is: A method comprising identifying that the video is a video that is played repeatedly based on the information regarding whether the video is played repeatedly.

Images