Electronic device and operation method thereof

Abstract

An electronic device according to the present disclosure comprises: a decoder for acquiring an image by decoding an input stream; a frames per second (FPS) conversion unit for converting an FPS of the decoded image; a display unit for outputting the image; a memory for storing one or more instructions; and at least one processor including processing circuitry, wherein, when the one or more instructions are individually or collectively executed by the at least one processor, the decoder, under control of the processor, when the FPS of the input stream is changed, can adjust the number of frame repetitions and acquire an image on the basis of a changed frame acquisition pattern corresponding to an FPS of the input stream before the change and an FPS of the input stream after the change, the FPS conversion unit, under control of the processor, can convert the FPS of the image acquired by adjusting the number of frame repetitions, and the display unit, under control of the processor, can output the image of which the FPS has been converted by the FPS conversion unit.

Description

Electronic device and method of operation thereof

[0001] The various disclosed embodiments relate to an electronic device and a method of operating the same, and more specifically, to an electronic device for removing motion judder of an input image and a method of operating the same.

[0002] Motion judder is a phenomenon in which video movement does not appear continuous and smooth, but rather appears choppy or shaky; it primarily occurs when playing videos with a low frame rate. To eliminate motion judder, products such as TVs and monitors perform a frame rate conversion operation. This conversion operation can be a method of outputting a high-frame-rate video by adding interpolated frames, generated using frame interpolation technology for low-frame-rate input video, between the input frames. Motion judder can be eliminated through this method.

[0003] An electronic device according to an embodiment may include a decoder that decodes an input stream to acquire an image, a frame rate converter that converts the frame rate (Frames Per Second, FPS) of the decoded image, a display unit that outputs the image, a memory that stores one or more instructions, and at least one processor including a processing circuit.

[0004] In an embodiment, when the one or more instructions are executed individually or collectively by at least one processor, the decoder can acquire the image by adjusting the number of frame repetitions based on a changed frame acquisition pattern corresponding to the frame rate of the input stream before the change and the frame rate of the input stream after the change, when the frame rate of the input stream is changed according to the control of the processor.

[0005] In an embodiment, when the one or more instructions are executed individually or collectively by at least one processor, the frame rate converter can convert the frame rate of the acquired image by adjusting the number of frame repetitions according to the control of the processor.

[0006] In an embodiment, when the one or more instructions are executed individually or collectively by at least one processor, the display unit can output an image in which the frame rate is converted by the frame rate converter under the control of the processor.

[0007] Figure 1 is a diagram illustrating the frame output timing when converting a 60 FPS input stream into a 120 FPS output image when converting the frame rate based on frame interpolation technology.

[0008] Figure 2 is a diagram illustrating the frame output timing when converting a 30 FPS input stream into a 120 FPS output video when converting the frame rate based on frame interpolation technology.

[0009] Figure 3 is a diagram illustrating the frame output timing when converting a 30 FPS input stream into a fixed FPS video and then acquiring a 120 FPS output video when converting the frame rate based on frame interpolation technology.

[0010] Figure 4 is a diagram illustrating the frame output timing when converting an input that changes from 30 FPS to 60 FPS to 120 FPS when converting the frame rate based on frame interpolation technology.

[0011] Figure 5 illustrates a method of applying a long delay regardless of the original FPS to resolve frame delay changes when the original FPS changes.

[0012] FIG. 6 is a block diagram of an electronic device according to an embodiment.

[0013] Figure 7 is a diagram for explaining the image processing unit of Figure 6 in more detail.

[0014] FIG. 8 illustrates a case where the encoder converts the original FPS to a fixed FPS according to an embodiment.

[0015] FIG. 9 is a block diagram of a frame rate conversion unit according to an embodiment.

[0016] FIG. 10 is an internal block diagram of an electronic device according to an embodiment.

[0017] FIG. 11 is a diagram illustrating, according to an embodiment, that when the FPS of an input stream changes, the decoder adjusts the number of frame repetitions and transmits them to the frame rate converter, thereby allowing frames to be output from the frame rate converter at regular intervals.

[0018] FIG. 12 is a diagram illustrating how, according to an embodiment, when the FPS of an input stream changes, the decoder adjusts the number of frame repetitions so that frames are output from the frame rate converter at regular intervals.

[0019] FIG. 13 is a diagram illustrating how, according to an embodiment, when the FPS of an input stream changes, the decoder adjusts the number of frame repetitions so that frames are output from the frame rate converter at regular intervals.

[0020] FIG. 14 is a diagram illustrating how, according to an embodiment, when the FPS of an input stream changes, the decoder adjusts the number of frame repetitions so that frames are output from the frame rate converter at regular intervals.

[0021] FIG. 15 is a flowchart illustrating the operation method of an electronic device according to an embodiment.

[0022] The method of operation of an electronic device according to an embodiment may include the step of decoding an input stream to acquire an image.

[0023] The method of operation of an electronic device according to an embodiment may include a step of converting the frame rate (Frames Per Second, FPS) of the decoded image.

[0024] In an embodiment, the step of acquiring the image may include the step of acquiring the image by adjusting the number of frame repetitions based on a changed frame acquisition pattern corresponding to the frame rate of the input stream before the change and the frame rate of the input stream after the change when the frame rate of the input stream is changed.

[0025] The method of operation of an electronic device according to an embodiment may include a step of converting the frame rate of the image obtained by adjusting the number of frame repetitions.

[0026] The method of operation of an electronic device according to an embodiment may further include the step of outputting an image in which the frame rate has been converted.

[0027] A recording medium according to an embodiment may be a computer-readable recording medium having a program capable of executing by a computer a method of operation of an electronic device, wherein the recording medium includes the step of decoding an input stream to acquire an image and the step of converting the frame rate (Frames Per Second, FPS) of the decoded image, and the step of acquiring the image includes, when the frame rate of the input stream is changed, the step of acquiring the image by adjusting the number of frame repetitions based on a change frame acquisition pattern corresponding to the frame rate of the input stream before change and the frame rate of the input stream after change, and the method further includes the step of converting the frame rate of the image acquired by adjusting the number of frame repetitions and the step of outputting the image with the converted frame rate.

[0028] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0029] Embodiments of the present disclosure are described below in detail with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0030] The terms used in this disclosure are described in their current, general form considering the functions mentioned herein; however, they may refer to various other terms depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Accordingly, the terms used in this disclosure should not be interpreted solely by their names, but should be interpreted based on the meaning of the terms and the overall content of this disclosure.

[0031] Furthermore, the terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure.

[0032] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other components in between.

[0033] The terms "above" and similar designations used in this specification, particularly in the claims, may indicate both singular and plural forms. Furthermore, unless there is a description explicitly specifying the order of the steps describing the method according to this disclosure, the described steps may be performed in a suitable order. This disclosure is not limited by the order in which the described steps are described.

[0034] Phrases such as "in some embodiments" or "in one embodiment" appearing in various places in this specification do not necessarily refer to the same embodiment.

[0035] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0036] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0037] Additionally, terms such as "...part," "module," etc., as described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0038] Additionally, the term "user" in the specification refers to a person using the electronic device and may include consumers, evaluators, viewers, administrators, or installers.

[0039] The present disclosure will be described in detail below with reference to the attached drawings.

[0040] Figure 1 is a diagram illustrating the frame output timing when converting a 60 FPS input stream into a 120 FPS output image when converting the frame rate based on frame interpolation technology.

[0041] Frame interpolation requires information on at least two frames, such as the previous and subsequent frames based on the interpolation ratio. Consequently, the output image is delayed by a certain number of frames relative to the input image. For example, to create an output frame at time point 1 that serves as an intermediate point between time point 0 and time point 2, data for both the time point 0 frame and the time point 2 frame are required. Since the interpolated frame at time point 1 is output after the time point 2 frame is input, a delay occurs between the time when the frame is input and the time when it is output.

[0042] Table 1 (10) of FIG. 1 shows the frame input and output timing when the frame rate converter converts the original 60 FPS input to the output 120 FPS.

[0043] In Table 1 (10), the original FPS indicates how many frames per second the original video is input into the frame buffer. Table 1 (10) shows the case where the original FPS is 60 FPS, which means that 60 frames of the original video are input into the frame buffer per second.

[0044] The input frame time indicates the time when each input frame is stored in the frame buffer. In Table 1 (10), the input frame time is displayed as an integer based on the input FPS for easier representation. For example, in Table 1 (10), the time points corresponding to 0, 1 / 60 second, 2 / 60 second, 3 / 60 second, … starting from the beginning of the video are represented as 0, 1, 2, 3, …, which indicates that the input frame is input at time 1, 2, 3, 4, … and stored in the frame buffer.

[0045] For the sake of convenience of explanation, the input frame time can also be interpreted as representing the frame number. That is, the input frame time being 1, 2, 3, and 4 can be thought of as the frames input at time 1, 2, 3, and 4 being frame 1, frame 2, frame 3, and frame 4, respectively.

[0046] The frame rate conversion unit loads two keyframes, a first keyframe and a second keyframe, from the frame buffer for frame interpolation and performs frame interpolation.

[0047] Here, the first keyframe may be a frame entered into the frame buffer before the second keyframe among the keyframes used for frame interpolation. The second keyframe may be the most recent keyframe entered after the first keyframe has been entered.

[0048] The interpolation ratio is a numerical value indicating which point in time between the first keyframe and the second keyframe the output / interpolated frame corresponds to. The interpolation ratio can have a value within the numerical range from 0 to 1. For example, if the interpolation ratio is 0, the output frame has the point in time of the first keyframe; if the interpolation ratio is 1, the output frame has the point in time of the second keyframe; and if the interpolation ratio is 0.5, the output frame has a point in time between the first keyframe and the second keyframe. When the output frame has a point in time between the first keyframe and the second keyframe, the output frame is generated by interpolating the first keyframe and the second keyframe in a 1:1 ratio.

[0049] The output frame point indicates the point in time or the frame number at which the output frame generated by applying an interpolation ratio to the first key frame and the second key frame is output.

[0050] In Table 1 (10), the part of reference numeral 11 is explained as follows: when the input frame time point to the frame rate converter is 4, that is, when the 4th frame is input into the frame buffer, the frame rate converter obtains two output frames by using the 2nd frame and the 3rd frame input before the 4th frame as key frames. The 3rd frame, which is the key frame input immediately before the 4th frame is input, is used as the 2nd key frame, and the 2nd frame, which is input before the 3rd frame, can be used as the 1st key frame. The frame rate converter can output the two output frames generated from the 2nd frame and the 3rd frame at times 2.5 and 3, respectively. Since two output frames are output while one input frame enters the frame buffer, 60 FPS is converted to 120 FPS by the frame rate converter.

[0051] Graph 1 (15) of FIG. 1 is a graph that compares the frame delay between the input frame time and the output frame time of Table 1 (10). In Graph 1 (15), the x-axis represents time, and the y-axis represents the frame number or frame time.

[0052] In Table 1 (10), it can be seen that the input frame is input once every 1 / 60 second, and the output frame is output once every 1120 seconds.

[0053] In Table 1 (10) and Graph 1 (15), if we denote the time when the 4th frame begins to be input into the frame buffer as t1 and the time when the 4th frame begins to be output from the frame rate converter as t2, it can be seen that from the time when the 4th frame begins to be input into the frame buffer until the 4th frame begins to be output from the frame rate converter, there is a delay of 1.5 frames at 60 FPS and 3 frames at 120 FPS. In this case, the delay between input and output can be expressed as 1.5 frames at input FPS or 3 frames at output FPS.

[0054] Figure 2 is a diagram illustrating the frame output timing when converting a 30 FPS input stream into a 120 FPS output video when converting the frame rate based on frame interpolation technology.

[0055] Table 2 (20) shows the frame timing when converting the original 30 FPS input to the output 120 FPS. As in Table 1 (10), since the timing of the video with an original FPS of 30 is expressed as an integer in 1 / 60 second increments, the interval between the input frame timings is 2, not 1. In addition, to generate output frame timings that increase by 0.5, which is 1 / 4 of the input frame timing interval of 2, the interpolation ratio increases by 0.25, not 0.5.

[0056] When one frame is input to the frame rate converter, four frames are output, so 30 FPS is converted to 120 FPS.

[0057] Looking at the part of reference numeral 21 in Table 2 (20), it can be seen that input frame 4 begins to be input into the frame buffer when the time point is 4, and the frame rate converter acquires frames at intervals of 0.25 using frame 0 and frame 2 that were input before frame 4. The frame rate converter outputs four frames at times 0.5, 1, 1.5, and 2.

[0058] Graph 2 (25) of FIG. 2 is a graph showing the frame delay between the input frame time and the output frame time of Table 2 (20).

[0059] In Table 2 (20) and Graph 2 (25), if we denote the time when the 4th frame begins to be input into the frame buffer as t3 and the time when the 4th frame begins to be output as t4, it can be seen that from the time when the 4th frame begins to be input into the frame buffer until the 4th frame begins to be output from the frame rate converter, there is a delay of 1.75 frames at 30 FPS and 7 frames at 120 FPS. Therefore, the frame delay between input and output is 1.75 frames at the input FPS and 7 frames at the output FPS.

[0060] Comparing Figure 1 and Figure 2, it can be seen that the amount of delay between the input frame time and the output frame time varies depending on the frame rate of the input image when the frame rate of the output frame is the same. In other words, the greater the difference in FPS between the input frame and the output frame, the greater the amount of delay between the input frame and the output frame time.

[0061] Figure 3 is a diagram illustrating the frame output timing when converting a 30 FPS input stream into a fixed FPS video and then acquiring a 120 FPS output video when converting the frame rate based on frame interpolation technology.

[0062] In Fig. 3, the frame rate converter assumes the case where a fixed FPS video, such as 60 FPS, is input.

[0063] In Fig. 2, the frame rate converter receives an original 30 FPS video and converts it directly to an output of 120 FPS, whereas in Fig. 3, the decoder is assumed to restore an original 30 FPS input stream into a fixed FPS video of 60 FPS.

[0064] In FIG. 3, the decoder can perform a pull-down operation that repeats the 30 FPS input twice to restore the original 30 FPS input into a 60 FPS video. The decoder can restore the input video, which is input at a rate of one frame per 1 / 30 second, by repeating it twice.

[0065] The decoder can acquire a restored image with a fixed FPS of 60 FPS and transmit it to a frame rate converter. The frame rate converter receives the 60 FPS image restored by the decoder as input and can convert the 60 FPS image into a 120 FPS output image and output it.

[0066] Table 3 (30) and graph 3 (35) of FIG. 3 explain the frame output timing when the frame rate converter receives a 60 FPS video, in which a 30 FPS video is repeated twice, from a decoder and converts it into a 120 FPS output video.

[0067] When comparing Table 2 (20) and Graph 2 (25) of FIG. 2 described above with Table 3 (30) and Graph 3 (35) of FIG. 3, it can be seen that in FIG. 2, a frame is input only once during the same period of time to the frame rate converter, whereas in FIG. 3, a repeating frame is input twice during the same period of time to the frame rate converter.

[0068] In Fig. 2, it was necessary to wait until the time of t3 for the second frame to be fully input, but in Fig. 3, since the repeating frame is input, it can be seen that the time when the second frame arrives is accelerated to t6. Therefore, the time when the frame rate converter performs frame interpolation using the second frame input into the frame buffer together with the zero frame is also accelerated, thereby reducing frame delay.

[0069] If we denote the time when the 4th frame begins to be input into the frame buffer in Figure 3 as t3 and the time when the 4th frame begins to be output as t5, it can be seen that a delay of 2.5 frames based on 60 FPS and 5 frames based on 120 FPS occurs from the time when the 4th frame begins to be input into the frame buffer until the 4th frame begins to be output from the frame rate converter.

[0070] Compared to the case where the frame rate converter receives the original FPS video and converts it directly to an output of 120 FPS, as shown in Fig. 3, when the decoder restores the original FPS input stream to a higher FPS video and the frame rate converter converts the frame, the difference in viewpoint between the input frame and the output frame is reduced.

[0071] Figure 4 is a diagram illustrating the frame output timing when converting an input that changes from 30 FPS to 60 FPS to 120 FPS when converting the frame rate based on frame interpolation technology.

[0072] There are cases where the original FPS is converted within the same content due to reasons such as the transmission environment or device settings.

[0073] Table 4 (40) and graph 4 (45) of FIG. 4 show the frame output time when a video with an original FPS of 30 FPS is input, and then a video with an original FPS of 60 FPS is input.

[0074] Referring to Table 4 (40) and Graph 4 (45), it can be seen that the original FPS is 30 FPS up to the 4th frame, and the original FPS changes to 60 FPS from the 6th frame onwards.

[0075] In the section where the original FPS is 30 FPS, the viewpoint interval between input frames is 2, so a frame is input every 2 intervals, and in the section where the original FPS is 60 FPS, the viewpoint interval between input frames is 1, so a frame is input every 1 interval.

[0076] When the original FPS is 30, the interpolation rate for frame rate conversion is performed at intervals of 0.25, and from the 6th frame, when the original FPS changes to 60, which is the 1st keyframe, the interpolation rate for frame rate conversion is switched at intervals of 0.5 to match the changed FPS.

[0077] At this time, the output frame timing is momentarily delayed when the original FPS changes. That is, the output frame timing increases by 0.5, and as shown in the parts indicated by reference numerals 41 and 42, the output timing increases by 2 from 4.5 to 6.5 in just two output frames, and thereafter it can be seen that it proceeds at a normal speed of 0.5 per frame.

[0078] This phenomenon occurs because the delay time required for frame interpolation differs depending on whether the source FPS is 30 FPS or 60 FPS. In other words, since the delay time between the input frame and the output frame is shorter when the source FPS is 60 FPS than when it is 30 FPS, momentary stuttering, known as motion interruption, occurs in the output video at the moment the output frame delay time shortens as the source FPS changes from 30 FPS to 60 FPS. In this case, to the user watching the video, the movement of the video appears to momentarily speed up or skip.

[0079] Motion stuttering is not limited to cases where the original FPS changes from 30 FPS to 60 FPS, but occurs whenever the original FPS changes from a low FPS to a high FPS, resulting in a shorter delay between the input and output images.

[0080] Conversely, when the FPS changes from a high FPS to a low FPS, such as when the original FPS changes from 60 FPS to 30 FPS, the delay time of the output frame increases with the change in the original FPS. In this case, to the user watching the video, the movement of the video appears to slow down momentarily or as if the movement has paused.

[0081] Figure 5 illustrates a method of applying a long delay regardless of the original FPS to resolve frame delay changes when the original FPS changes.

[0082] Referring to Table 5 (50) and Graph 5 (55) in FIG. 5, the frame rate converter can prevent motion stuttering by matching the delay time between the input frame and the output frame when the original FPS is 60 FPS to the delay time between the input frame and the output frame when the original FPS is 30 FPS.

[0083] Since the delay between an input frame and an output frame is longer when the original FPS is 30 than when it is 60, the difference in delay time can be eliminated by increasing the delay at the original FPS of 60 to match the delay at the original FPS of 30. However, doing so means that as the frame delay increases, the delay interval between the input frame and the output frame becomes longer, and consequently, more frame buffer is required.

[0084] For example, when compared to Table 4 (40) in FIG. 4, unlike in Table 4 (40) in FIG. 4 where interpolation is performed using frames 6 and 7 when frame 8 is input, in Table 5 (50) in FIG. 5, frames 6 and 7 must be used when frame 9 is input instead of frame 8, so frame 8 must be stored in a separate buffer. Therefore, the frame rate conversion unit requires a separate storage space to store the third key frame, and the total number of buffers required increases.

[0085] In addition, if the original FPS is lower than 30 FPS, an additional frame buffer of at least the fourth keyframe is required in the frame rate converter to ensure a seamless transition with the original FPS.

[0086] That is, according to Fig. 5, not only is the time difference between the input frame and the output frame increased, but an additional frame buffer must also be used to prevent motion stuttering, so memory usage increases due to the use of the additional buffer.

[0087] Accordingly, in a device that eliminates motion judder by converting a low-frame-rate input video into a high-frame-rate output video, a technology is required to generate an output video without motion interruption or motion stoppage even when the frame rate of the input video changes.

[0088] In addition, given that increasing the delay between the input and output images increases memory usage, there is a need for a technology that outputs video without interruption of motion without increasing buffer or memory usage.

[0089] FIG. 6 is a block diagram of an electronic device (100) according to an embodiment.

[0090] Referring to FIG. 6, the electronic device (100) may include a processor (110), a memory (120), an image processing unit (130), and a display unit (140).

[0091] The electronic device (100) according to the embodiment may be an electronic device capable of outputting an image.

[0092] In an embodiment, the electronic device (100) may be a video display device capable of outputting an image or video through a display, and / or an audio device capable of outputting audio. The electronic device (100) may be fixed or mobile.

[0093] In an embodiment, the electronic device (100) may include at least one of a television, desktop, smartphone, tablet PC, game console, audio device, mobile phone, video phone, e-book reader, laptop PC, netbook computer, digital camera, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), camcorder, navigation, wearable device, smart watch, home network system, security system, and medical device.

[0094] If the electronic device (100) is an image display device, the electronic device (100) may be implemented as a flat display device, a curved display device having a screen with curvature, or a flexible display device with adjustable curvature. The output resolution of the electronic device (100) may have various resolutions, such as HD (High Definition), Full HD, Ultra HD, or a resolution sharper than Ultra HD.

[0095] The electronic device (100) can output various forms of content provided by content providers.

[0096] A content provider may refer to terrestrial, cable, or satellite broadcasters that provide various types of content to consumers, IPTV (Internet Protocol Television) service providers or OTT (Over the Top) service providers, or server operators that provide various kinds of content.

[0097] Content can be in various forms, such as video including still images or moving images, or audio, subtitles, and other supplementary information.

[0098] A memory (120) according to one embodiment can store various data, programs, or applications for driving and controlling an electronic device (100).

[0099] Memory (120) may store at least one program that is executed by the processor (110). The program stored in memory (120) may include one or more instructions. The program, one or more instructions, or application stored in memory (120) may be executed by the processor (110).

[0100] A predefined operation rule or AI model may be stored in the memory (120). Additionally, the memory (120) may store data that is input to or output from the electronic device (100).

[0101] The memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0102] A processor (110) according to an embodiment controls the overall operation of the electronic device (100) and the signal flow between the internal components of the electronic device (100), and performs the function of processing data.

[0103] The processor (110) may include a processing circuit.

[0104] The processor (110) may include a single core, dual core, triple core, quad core, and multiples thereof. The processor (110) may be one or may include multiple processors. For example, the processor (110) may be implemented as a main processor and a sub processor.

[0105] The processor (110) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a VPU (Video Processing Unit). Alternatively, according to an embodiment, the processor (110) may be implemented in the form of a System On Chip (SoC) integrating at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (110) may further include a Neural Processing Unit (NPU).

[0106] A processor (110) according to an embodiment can control the overall operation of an electronic device (100). The processor (110) can control the electronic device (100) to function by executing one or more instructions stored in memory (120).

[0107] In an embodiment, at least one processor (110) can control the operation of an image processing unit (130) by executing one or more instructions stored in memory (120) individually or collectively.

[0108] In an embodiment, the image processing unit (130) is controlled by at least one processor (110) to signal process an input image and obtain an output image.

[0109] In an embodiment, the image processing unit (130) can be controlled by at least one processor (110) to process the input image so that motion judder is removed.

[0110] In an embodiment, the image processing unit (130) may include a decoder (133) and a frame rate conversion unit (134).

[0111] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can obtain an image by decoding an input stream under the control of the processor (110). The image obtained by the decoder (133) may also be referred to as a restored image.

[0112] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) acquires a changed frame acquisition pattern corresponding to the frame rate of the input stream before the change and the frame rate of the input stream after the change when the frame rate of the input stream is changed, and can acquire an image by adjusting the number of frame repetitions based on the changed frame acquisition pattern.

[0113] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire a restored image by adjusting the number of frame repetitions based on the FPS of the input stream before the change and the FPS of the input stream after the change, when the FPS of the input stream changes according to the control of the processor (110).

[0114] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can obtain FPS information of the input stream from the header of the input stream under the control of the processor (110).

[0115] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can identify that the FPS of the input stream has changed based on the FPS information of the input stream under the control of the processor (110).

[0116] In an embodiment, the decoder (133) assumes that it acquires a restored image having a fixed FPS under the control of the processor (110).

[0117] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire a restored image according to a frame acquisition pattern corresponding to an input stream under the control of the processor (110).

[0118] In an embodiment, the frame acquisition pattern corresponding to the input stream can be determined according to the FPS of the input stream and the fixed FPS of the decoder (133).

[0119] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire a restored image by adjusting the number of frame repetitions according to a changed frame acquisition pattern that is different from the frame acquisition pattern corresponding to the input stream after the change, when the FPS of the input stream is changed according to the control of the processor (110).

[0120] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can identify a change frame acquisition pattern from a matrix or mapping table stored in memory (120) under the control of the processor (110).

[0121] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can identify a change frame acquisition pattern according to the FPS of the input stream before change and the FPS of the input stream after change in a matrix or mapping table under the control of the processor (110).

[0122] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can, under the control of the processor (110), adjust the number of frame repetitions of the restored image by using a pull-down method selected according to the difference in frame rates between the input stream FPS and the fixed FPS as a frame acquisition pattern if the fixed FPS of the restored image restored by the decoder (133) is higher than the FPS of the input stream.

[0123] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire the restored image according to a first frame acquisition pattern corresponding to the first input FPS if the input FPS of the input stream is the first input FPS, under the control of the processor (110).

[0124] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire the restored image according to a second frame acquisition pattern corresponding to the second input FPS if the input FPS of the input stream is the second input FPS, under the control of the processor (110).

[0125] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire a restored image according to a changed frame acquisition pattern different from the second frame acquisition pattern for a predetermined time from the point in time when the FPS of the input stream changes from the first input FPS to the second FPS, under the control of the processor (110).

[0126] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire a restored image by increasing the number of repetitions of the repeated frames acquired according to the second frame acquisition pattern at the time when the FPS of the input stream changes from the first input FPS to the second FPS, under the control of the processor (110).

[0127] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the decoder (133) can acquire a restored image by skipping a repeating frame acquired according to a second frame acquisition pattern if the second input FPS is lower than the first input FPS, under the control of the processor (110).

[0128] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the frame rate converter (134) can receive FPS information of the restored image and input stream from the decoder (133) under the control of the processor (110).

[0129] In an embodiment, when one or more instructions stored in memory (120) are executed individually or collectively by at least one processor (110), the frame rate converter (134) can obtain an output image by adding an interpolated frame using the FPS information of the input stream and the fixed FPS of the decoder (133) under the control of the processor (110).

[0130] In the embodiment, the display unit (140) can output an image. In the embodiment, the display unit (140) can output the output image obtained by the frame rate conversion unit (134) to the screen in sequence. Thus, according to the embodiment, when the FPS of the input stream changes, the decoder (133) can obtain a restored image of a fixed FPS by adjusting the number of frame repetitions based on the FPS of the input stream before the change and the FPS of the input stream after the change, under the control of the processor (110). That is, unlike restoring the image by considering only the frame rate of the input stream after the change without considering the frame rate of the input stream before the change even when the FPS of the input stream changes, according to the embodiment, the number of frame repetitions can be adjusted by considering both the frame rate of the input stream before the change and the frame rate of the input stream after the change.

[0131] The decoder (133) can acquire an image by adjusting the number of frame repetitions based on a change frame acquisition pattern corresponding to the frame rate of the input stream before change and the frame rate of the input stream after change, and transmit it to the frame rate conversion unit (134).

[0132] Accordingly, the frame rate converter (134), which receives the restored image with the frame repetition count adjusted from the decoder (133), can obtain a smooth output image without motion interruption or motion stoppage from the restored image.

[0133] In FIG. 6, as an example embodiment, the decoder (133) and the frame rate converter (134) are described as operating under the control of the processor (110), but the present disclosure is not limited thereto.

[0134] In an embodiment, the decoder (133) and / or the frame rate converter (134) may each be implemented as dedicated hardware. For example, the decoder (13) and / or the frame rate converter (134) may be in the form of a hardware chip equipped with a circuit capable of directly performing the aforementioned operations. The decoder (13) and / or the frame rate converter (134) may be manufactured in the form of at least one hardware chip and mounted on an electronic device (100).

[0135] Hereinafter, in the present disclosure, the fact that the decoder (133) and the frame rate converter (134) perform a specific operation includes cases where the decoder (133) and the frame rate converter (134) are implemented in hardware and directly perform a specific operation, and cases where the decoder (133) and the frame rate converter (134) operate under the control of at least one processor (110).

[0136] FIG. 7 is a drawing for explaining the image processing unit (130) of FIG. 6 in more detail.

[0137] Referring to FIG. 7, the image processing unit (130) may include a decoder (133) and a frame rate conversion unit (134).

[0138] The decoder (133) according to the embodiment can receive an input stream having any original FPS.

[0139] In an embodiment, the decoder (133) can decode an input stream having an original FPS to obtain a restored image having a fixed FPS. In an embodiment, it is assumed that the fixed FPS of the decoder output image is equal to or higher than the original FPS.

[0140] In an embodiment, the restored image acquired by the decoder (133) may be referred to as the decoder output image. Additionally, the decoder output image may include decoder frames.

[0141] In an embodiment, the decoder (133) can receive an input stream having an arbitrary original FPS and, in the process of decoding, output a decoder output image having a fixed FPS such as 60 FPS and original FPS information.

[0142] In an embodiment, the decoder (133) may receive an input stream from an encoder or a server. The input stream contains compressed video data. In an embodiment, the decoder (133) may receive the input stream, decompress it, and convert it into individual frames.

[0143] In an embodiment, the decoder (133) can obtain original FPS information of the input stream from the header of the input stream. In an embodiment, the decoder (133) can obtain original FPS information of the input stream from the metadata included in the header of the input stream.

[0144] In the embodiment, the decoder (133) is configured to have a restored image with a fixed FPS, so the decoder (133) knows information about the fixed FPS.

[0145] In an embodiment, the decoder (133) can identify a frame acquisition pattern corresponding to the input stream by using the original FPS information of the input stream and the fixed FPS information of the restored image.

[0146] In the embodiment, the frame acquisition pattern may be a fixed rule used by the decoder (133) to acquire a restored image of a fixed FPS from an input stream having an original FPS.

[0147] In the embodiment, the frame acquisition pattern corresponding to the input stream can be determined according to the FPS and fixed FPS of the input stream. Since the fixed FPS of the decoder (133) has a predetermined value, the frame acquisition pattern can be determined according to the original FPS of the input stream.

[0148] In an embodiment, the frame acquisition pattern may include rules indicating whether to repeat frames to acquire a restored image, which frames to repeat how many times, etc.

[0149] In the embodiment, the frame acquisition pattern may be stored in memory (120) in the form of a matrix or a mapping table corresponding to the original FPS of the input stream.

[0150] In an embodiment, the decoder (133) can acquire a restored image according to a frame acquisition pattern corresponding to the input stream.

[0151] For example, if the input FPS of the input stream is the first input FPS, the decoder (133) can acquire a restored image according to the first frame acquisition pattern corresponding to the first input FPS.

[0152] For example, if the input FPS of the input stream is the second input FPS, the decoder (133) can acquire a restored image according to a second frame acquisition pattern corresponding to the second input FPS.

[0153] For example, if the original FPS and the fixed FPS of the restored image obtained from the decoder (133) are both 60 FPS, the decoder (133) does not need to repeat the frame, so the restored image can be obtained using a frame acquisition pattern that obtains the decoded frame as is.

[0154] In an embodiment, when the fixed FPS of the output image is higher than the original FPS, the frame acquisition pattern may include a pull-down method. A pull-down method is a method of repeatedly inserting specific frames to compensate for the difference in frame rates.

[0155] In an embodiment, the decoder (133) can select an appropriate pull-down method based on the difference in frame rates between the input stream's FPS and the fixed FPS.

[0156] In an embodiment, the decoder (13) can perform pull-down by repeatedly using keyframes.

[0157] In an embodiment, the decoder (133) can adjust the frame rate by selecting a key frame, which is a reference frame for frame rate conversion, and repeatedly inserting the key frame.

[0158] A keyframe refers to a frame selected from the original frames to be repeated or inserted. A repeated frame that is not a keyframe may be referred to as a "repeat frame." Repeat frames can be inserted between keyframes to adjust the FPS of the entire video.

[0159] In an embodiment, the decoder (133) may include a buffer that stores the decoded image inside the decoder (133). The decoder (133) may temporarily store frames in the buffer and output frames as many times as necessary.

[0160] In an embodiment, the decoder (133) may include two or more buffers. The decoder (133) may perform a pull-down operation in such a way that it performs decoding in one buffer, outputs the frame stored in the buffer when decoding is complete, and performs decoding in another buffer.

[0161] For example, if the original FPS is 24 FPS and the fixed FPS of the restored image acquired by the decoder (133) is 60 FPS, the decoder (133) may use a 3:2 pull-down method as the frame acquisition pattern. The decoder (133) may restore the first frame among the frames acquired from the input stream, temporarily store it in a buffer, and output the first frame stored in the buffer three times in succession. Additionally, the decoder (133) may restore the second frame, temporarily store it in a buffer, and output the second frame stored in the buffer two times in succession.

[0162] Alternatively, if the original FPS is 30 FPS and the fixed FPS of the restored image obtained by the decoder (133) is 60 FPS, the decoder (133) may use a 2:2 pulldown method as the frame acquisition pattern. The decoder (133) may obtain the restored image by repeating each frame of the input stream twice.

[0163] In the embodiment, the original FPS of the input stream input to the decoder (133) can be changed.

[0164] A server providing content can transmit video with different FPS to an electronic device (100) depending on the network speed. The electronic device (100) can request a video stream having an adjusted FPS from the server, taking into account the performance of the electronic device (100) or network performance based on network latency, and the server can transmit video with an FPS that matches the network performance to the electronic device (100).

[0165] Alternatively, the user may control the electronic device (100) to adjust the FPS requested from the server. The user may change the app settings of the electronic device (100) from 60 FPS to 30 FPS or vice versa. Accordingly, the electronic device (100) requests a stream of the changed FPS from the server and receives the video of the FPS requested by the electronic device (100) from the server.

[0166] When an electronic device (100) receives an input stream corresponding to specific content from a server and plays it, if it receives an input stream corresponding to the specific content from the server that has a different FPS than the previous one, the time difference between the input and output frames changes as described above, and a phenomenon such as motion interruption or motion stopping occurs.

[0167] In an embodiment, the decoder (133) can adjust the number of frame repetitions of the decoder output image obtained from the decoder (133) to offset the time delay difference between the input and output frames due to the change in the original FPS of the input stream when the original FPS of the input stream changes. Accordingly, it is possible to prevent the movement of the output image output from the frame rate conversion unit (134) from being interrupted or changing abruptly.

[0168] In the embodiment, the time delay between input / output frames may refer to the time / time difference between the time when a frame is input to the frame rate converter (134) and the time when a frame is output from the frame rate converter (134).

[0169] In the embodiment, the time at which a decoder frame is output from the decoder (133) can correspond to the time at which a decoder frame is input to the frame rate converter (134).

[0170] Accordingly, in the present disclosure, the time delay between input / output frames may mean the difference between the time when a decoder frame is acquired from the decoder (133) and the time when a frame rate conversion unit frame is output from the frame rate conversion unit (134).

[0171] In the present disclosure, the time delay between input / output frames may also be referred to as the difference in input / output timing.

[0172] In an embodiment, the decoder (133) can identify whether the original FPS of the input stream has changed. In an embodiment, the decoder (133) can obtain the original FPS information of the input stream from the header of the input stream.

[0173] In the embodiment, the original FPS information is information indicating how many FPS per second of the input stream is, and may be information indicating the frame rate of the video. FPS stands for Frames Per Second and represents the frame rate. The frame rate refers to the number of frames per second and may represent the number of image frames displayed on the screen during one second.

[0174] The original FPS information can be included in the video file header. The video file header contains various metadata information necessary for video playback, and this metadata information may include the original FPS information of the input stream.

[0175] In an embodiment, the decoder (133) can identify, based on the original FPS information, that the FPS of the input stream has changed to a different FPS than the FPS of the previous input stream.

[0176] In an embodiment, when the decoder (133) identifies that the FPS of the input stream has changed, it can obtain a restored image by adjusting the number of frame repetitions based on the original FPS of the input stream before the change and the original FPS of the input stream after the change.

[0177] That is, in the embodiment, when the FPS of the input stream changes, the decoder (133) can adjust the number of frame repetitions by considering not only the FPS of the input stream after the change but also the FPS of the input stream before the change.

[0178] In an embodiment, the decoder (133) can identify a change frame acquisition pattern based on the FPS of the input stream before change and the FPS of the input stream after change.

[0179] In the embodiment, the change frame acquisition pattern is similar to the frame acquisition pattern in that it is a rule used by the decoder (133) to acquire the restored image, but it can be distinguished from the frame acquisition pattern in that it is a pattern used at the time when the original FPS of the input stream changes, unlike a general frame acquisition pattern.

[0180] In the embodiment, the frame acquisition pattern corresponding to the input stream is determined according to the FPS of the currently input stream, whereas the change frame acquisition pattern can be distinguished from the frame acquisition pattern in that it is determined by considering both the FPS of the input stream before change and the FPS of the input stream after change.

[0181] In an embodiment, the decoder (133) can acquire a restored image by adjusting the number of frame repetitions according to a changed frame acquisition pattern that is different from the frame acquisition pattern corresponding to the input stream after the change when the FPS of the input stream changes.

[0182] In an embodiment, a change frame acquisition pattern may be stored in the memory (120). In an embodiment, the change frame acquisition pattern corresponding to the original FPS of the input stream before change and the original FPS of the input stream after change may be stored in the memory (120) as a set in the form of a matrix or a mapping table.

[0183] In an embodiment, the decoder (133) can identify a change frame acquisition pattern corresponding to the original FPS of the input stream before change and the original FPS of the input stream after change from a matrix or mapping table stored in memory (120).

[0184] In an embodiment, the decoder (133) can acquire a restored image by adjusting the number of frame repetitions using an identified change frame acquisition pattern.

[0185] In an embodiment, the decoder (133) can acquire a restored image using a changed frame acquisition pattern for a predetermined time from the point in time when the FPS of the input stream is changed, and then identify a frame acquisition pattern corresponding to the changed input stream and acquire a restored image based thereon.

[0186] For example, it is assumed that the first input stream has a source FPS of the first FPS, and the frame acquisition pattern corresponding to the first input stream when the source FPS is the first FPS is the first frame acquisition pattern, and the second input stream has a source FPS of the second FPS, and the frame acquisition pattern corresponding to the second input stream when the source FPS is the second FPS is the second frame acquisition pattern. Here, it is assumed that the first input stream and the second input stream are streams for the same content.

[0187] In an embodiment, the decoder (133) can obtain a restored image by processing a first input stream, in which the original FPS is the first FPS, according to a first frame acquisition pattern.

[0188] In an embodiment, when a first input stream is input and then a second input stream, which is another input stream of the same content, is input, the decoder (133) identifies that the FPS of the input stream changes from the first input FPS to the second FPS and can identify a change frame acquisition pattern corresponding to the first input FPS before the change and the second input FPS after the change.

[0189] In an embodiment, the decoder (133) can cancel out the change in viewpoint delay of the output image output from the frame rate conversion unit (134) by decoding the second input stream according to the change frame acquisition pattern to acquire the restored image and transmitting it to the frame rate conversion unit (134).

[0190] Afterwards, the decoder (133) can obtain a restored image by decoding the second input stream according to a second frame acquisition pattern corresponding to the second input FPS, corresponding to the second input FPS of the second input stream, which is a changed input stream.

[0191] In an embodiment, when the FPS of the input stream changes from the first input FPS to the second FPS, if the second input FPS is higher than the first input FPS, that is, if the second FPS is a higher frame rate than the first FPS, the decoder (133) can acquire a restored image by increasing the number of repetitions of the repeated frames acquired according to the second frame acquisition pattern.

[0192] In an embodiment, the decoder (133) can acquire a restored image by skipping the repeated frames acquired according to the second frame acquisition pattern when the FPS of the input stream changes from the first input FPS to the second FPS, that is, when the second input FPS is lower than the first input FPS, i.e., when the second FPS is a lower frame rate than the first FPS.

[0193] In an embodiment, the decoder (133) can transmit the original FPS information of the restored image and input stream to the frame rate converter (134).

[0194] In an embodiment, the frame rate converter (134) receives original FPS information of the restored image and input stream from the decoder (133), and can obtain an output image by adding an interpolated frame using the original FPS information and the fixed FPS of the decoder (133). The image output by the frame rate converter (134) may be referred to as the frame rate converter output image so as to be distinguished from the decoder output image output by the decoder (133).

[0195] In this way, according to the embodiment, when the FPS of the input stream changes, the decoder (133) can adjust the number of frame repetitions by using a combination of the original FPS of the input stream before change and the original FPS of the input stream after change, rather than a frame acquisition pattern corresponding to the FPS of the input stream currently being input, thereby eliminating the delay in the output time of the frame rate conversion unit (134).

[0196] FIG. 8 illustrates a case where the encoder (800) converts the original FPS to a fixed FPS according to an embodiment.

[0197] Referring to FIG. 8, the encoder (800) according to the embodiment can generate a video having a fixed FPS from the time the video is generated.

[0198] In an embodiment, when the encoder (800), rather than the decoder (133), encodes the video, a pull-down operation may be performed so that the video with the original FPS becomes an input stream with a fixed FPS.

[0199] In an embodiment, the encoder (800) can make the FPS of the video a fixed FPS by performing an appropriate pull-down using the original FPS and the fixed FPS. For example, if the original FPS is 24 FPS and the fixed FPS is 60 FPS, the encoder (800) can generate an input stream having a fixed FPS using a 3:2 pull-down method.

[0200] Alternatively, if the original FPS is 30 FPS and the fixed FPS is 60 FPS, the encoder (800) can generate an input stream with a fixed FPS using a 2:2 pull-down method.

[0201] In an embodiment, the encoder (800) can transmit an input stream having a fixed FPS and original FPS information to the decoder (133) through a communication network.

[0202] In an embodiment, the decoder (133) can receive an input stream having a fixed FPS and original FPS information from the encoder (800) through a communication network.

[0203] In the embodiment, since the decoder (133) receives an input stream having a fixed FPS, there is no need to convert the FPS, so the decoder output image can be obtained by performing only decoding without converting the FPS.

[0204] In an embodiment, the decoder (133) can transmit the decoder output image and original FPS information to the frame rate converter (134).

[0205] In an embodiment, the frame rate conversion unit (134) can convert the frame rate of a decoder output image having a fixed FPS using original FPS information so that frame rate conversion unit frames are output at regular intervals.

[0206] FIG. 9 is a block diagram of a frame rate conversion unit (134) according to an embodiment.

[0207] Referring to FIG. 9, the frame rate conversion unit (134) may include a key frame extraction unit (910), a frame buffer (920), a motion estimation unit (930), and a frame interpolation unit (940).

[0208] In an embodiment, the frame rate converter (134) can receive a decoder output image having a fixed FPS and original FPS information from the decoder (133).

[0209] In an embodiment, the frame rate conversion unit (134) can convert a decoder output image having a fixed FPS into an output image having an output FPS using the original FPS and output it. In order to distinguish it from the decoder output image output from the decoder (133), the output image output from the frame rate conversion unit (134) may be referred to as the frame rate conversion unit output image.

[0210] In an embodiment, the key frame extraction unit (910) can extract only key frames from the restored image received from the decoder (133). Key frames may refer to frames corresponding to the original FPS, rather than repeat frames added by the pull-down operation.

[0211] In an embodiment, the keyframe extraction unit (910) can extract keyframes from the restored video according to the original FPS information using the original FPS information. For example, in the case where the original FPS is 24 FPS and the restored video is a 60 FPS video generated by 3:2 pulldown, the keyframe extraction unit (910) can select 2 keyframes out of 5 inputs by extracting the first of three repeated inputs from the frames of the restored video, and then extracting the first of two repeated inputs. Thus, 24 keyframes are extracted from the 60 FPS inputs, and the video has the same FPS as the original video.

[0212] In an embodiment, the keyframe extraction unit (910) can extract only the keyframes from the input image and store them in the frame buffer (920).

[0213] In an embodiment, the frame buffer (920) may be a space for temporarily storing decoded frames. In an embodiment, the frame buffer (920) may be configured as a dedicated memory that can be accessed at high speed for high-speed data processing. That is, the frame buffer (920) may be a dedicated frame buffer memory separate from the memory (120) of FIG. 6. However, it is not limited thereto, and depending on the design and configuration of the electronic device (100), the frame buffer (920) may share a portion of the memory (120) of FIG. 6.

[0214] In an embodiment, the frame buffer (920) may include three separate buffers. The first of the three buffers may be a buffer in which the keyframe currently acquired by the keyframe extraction unit (910) is stored. When the keyframe extraction unit (910) stores a newly acquired keyframe in the first buffer, the keyframe stored in the first buffer may be moved to the second frame buffer. In this case, the keyframe stored in the second frame buffer is moved to the third buffer. That is, in order to store a new keyframe in the buffer, the data on the buffer is continuously changed. However, the effect of changing the data may be achieved by changing the memory address pointed to by each frame buffer without actually changing the data on the buffer.

[0215] Here, among the keyframes used for frame interpolation, the most recent keyframe that was stored in the first buffer and moved to the second buffer is referred to as the second keyframe, and the keyframe that was stored in the second buffer prior to the second keyframe and moved to the third buffer upon the input of the second keyframe is referred to as the first keyframe.

[0216] In an embodiment, the motion estimation unit (930) obtains a first keyframe and a second keyframe from the frame buffer (920) and can estimate motion information, such as a motion vector required for frame interpolation, from the first keyframe and the second keyframe.

[0217] In an embodiment, the motion estimation unit (930) may transmit the motion vector back to the frame buffer (920). However, it is not limited thereto, and the motion estimation unit (930) may also transmit the motion vector directly to the frame interpolation unit (940).

[0218] In an embodiment, the frame interpolation unit (940) can output an intermediate frame according to the first keyframe and the second keyframe, and the motion vector and interpolation ratio calculated from the two frames.

[0219] For example, if the original FPS of the input stream is 60, the fixed FPS of the decoder output video is 60, and the FPS of the frame rate conversion unit output video is 120, the keyframe extraction unit (910) can extract all 60 FPS video as keyframes and store them in the frame buffer (920).

[0220] In the embodiment, the frame interpolation unit (940) can determine the interpolation ratio according to the original FPS. That is, in the above example, the frame interpolation unit (940) can perform frame rate conversion by adding one interpolated frame with an interpolation ratio of 0.5 between each keyframe using the keyframes stored in the frame buffer (920).

[0221] For example, if the original FPS is 30, the fixed FPS of the decoder output video is 60, and the FPS of the frame rate conversion unit output video is 120, the key frame extraction unit (910) can remove repeated frames by pull-down from the decoder output video having a fixed 60 FPS and extract a key frame of 30 FPS, which is the same as the original, and store it in the frame buffer (920).

[0222] In this case, the frame interpolation unit (940) can obtain a key frame of 30 FPS from the frame buffer (920) and perform frame rate conversion by adding three interpolation frames with interpolation ratios of 0.25, 0.5, and 0.75 to the key frame.

[0223] In the embodiment, when the original FPS of the input stream changes, the decoder (133) can adjust and acquire the number of frames according to the change frame acquisition pattern and transmit them to the frame rate conversion unit (134).

[0224] In an embodiment, the frame rate converter (134) receives a controlled number of decoder frames from the decoder (133) at the time when the original FPS changes, and accordingly, can store a key frame in the frame buffer (920).

[0225] In the embodiment, when the original FPS of the input stream changes, the decoder (133) acquires a frame adjusted according to the change frame acquisition pattern and transmits it to the frame rate converter (134), so the frame rate converter (134) performs frame interpolation using the changed frame as the first key frame and the second key frame.

[0226] For example, if the original FPS of the input stream increases, the frame rate converter (134) receives additional frames from the decoder (133), causing the frame input to be delayed; conversely, if the original FPS of the input stream decreases, the frame rate converter (134) receives the next frame at the next point in time early from the decoder (133), and accordingly, the key frame used by the frame rate converter (134) for frame interpolation is also changed.

[0227] Accordingly, the frame rate conversion unit (134) can maintain a constant time delay between input and output frames by performing frame interpolation using the adjusted number of frames input from the decoder (133) at the time when the original FPS changes.

[0228] FIG. 10 is an internal block diagram of an electronic device (100) according to an embodiment.

[0229] The electronic device (100) of FIG. 10 may be an example of the electronic device (200) of FIG. 6.

[0230] Referring to FIG. 10, the electronic device (100) may include a tuner unit (1010), a communication unit (1020), a sensing unit (1030), an input / output unit (1040), a video processing unit (1050), a display unit (140), an audio processing unit (1060), an audio output unit (1070), and a user interface (1080) in addition to a processor (110) and a memory (120).

[0231] The tuner unit (1010) can select only the frequency of the channel to be received by the electronic device (100) from among many radio wave components by tuning through amplification, mixing, resonance, etc. of broadcast content received via wired or wireless connection. The content received through the tuner unit (1010) is decoded and separated into audio, video, and / or additional information. The separated audio, video, and / or additional information can be stored in memory (120) under the control of the processor (110).

[0232] The communication unit (1020) can connect the electronic device (100) to an external device or server under the control of the processor (110). The electronic device (100) can download programs or applications required by the electronic device (100) from an external device or server, etc., or perform web browsing through the communication unit (1020). In addition, the communication unit (1020) can receive content from an external device.

[0233] In an embodiment, the external device may include an encoder (800) shown in FIG. 8.

[0234] The communication unit (1020) may include at least one of a wireless LAN (1021), Bluetooth (1022), and wired Ethernet (1023) in accordance with the performance and structure of the electronic device (100). The communication unit (1020) may receive a control signal through a control device (not shown), such as a remote control, under the control of the processor (110). The control signal may be implemented as a Bluetooth type, an RF signal type, or a Wi-Fi type. The communication unit (1020) may further include other short-range communication (e.g., NFC (near field communication, not shown), BLE (Bluetooth Low Energy, not shown)) in addition to Bluetooth (1022). According to an embodiment, the communication unit (1020) may transmit and receive a connection signal to an external device, etc., through short-range communication such as Bluetooth (1022) or BLE.

[0235] The detection unit (1030) detects the user's voice, the user's image, or the user's interaction and may include a microphone (1031), a camera unit (1032), and an optical receiver (1033). The microphone (1031) can receive the user's uttered voice and convert the received voice into an electrical signal to output to the processor (110). The camera unit (1032) includes a sensor (not shown) and a lens (not shown) and can capture an image formed on a screen. The optical receiver (1033) can receive an optical signal (including a control signal). The optical receiver (1033) can receive an optical signal corresponding to user input (e.g., touch, press, touch gesture, voice, or motion) from a control device (not shown), such as a remote control or a mobile phone. A control signal can be extracted from the received optical signal under the control of the processor (110).

[0236] The input / output unit (1040) can receive additional information such as video (e.g., video signal or still image signal), audio (e.g., voice signal or music signal), and metadata from a device outside the electronic device (100) under the control of the processor (110). The input / output unit (1040) may include one of an HDMI port (High-Definition Multimedia Interface port, 1041), a component jack (1042), a PC port (1043), and a USB port (1044). The input / output unit (1040) may include a combination of the HDMI port (1041), the component jack (1042), the PC port (1043), and the USB port (1044).

[0237] The video processing unit (1050) processes video data to be displayed by the display unit (140) and can perform various video processing operations such as decoding, rendering, scaling, noise filtering, frame rate conversion, and resolution conversion on the video data.

[0238] In the embodiment, the video processing unit (1050) can perform the function of the image processing unit (130) of FIG. 6.

[0239] In an embodiment, the video processing unit (1050) can decode the input stream to obtain a restored image having a fixed FPS.

[0240] In an embodiment, the video processing unit (1050) can obtain a restored image by adjusting the number of frame repetitions based on the FPS of the input stream before the change and the FPS of the input stream after the change when the FPS of the input stream changes while decoding the input stream to obtain a restored image.

[0241] In an embodiment, when the communication unit (1020) receives an input stream from the encoder (800) shown in FIG. 8 in which the number of frame repetitions has already been adjusted in correspondence with the change in FPS of the input stream, the video processing unit (1050) can obtain a restored image by decoding the input stream without needing to consider the change in FPS of the input stream.

[0242] In an embodiment, the display unit (140) can output an image. In an embodiment, the display unit (140) can receive an image processed by the video processing unit (1050) and output it.

[0243] In an embodiment, the display unit (1055) can output content received from a broadcasting station, an external server, or an external storage medium to the screen. The content is a media signal and may include a video signal, an image, a text signal, etc. Additionally, the display unit (1055) can display a video signal or an image received through the HDMI port (1041) on the screen.

[0244] When the display unit (140) is implemented as a touch screen, the display unit (140) can be used as an input device in addition to an output device. Also, depending on the implementation form of the electronic device (100), the electronic device (100) may include two or more display units (140).

[0245] The audio processing unit (1060) performs processing on audio data. Various processing such as decoding, amplification, and noise filtering on the audio data can be performed in the audio processing unit (1060).

[0246] The audio output unit (1070) can output audio included in content received through the tuner unit (1010) under the control of the processor (110), audio input through the communication unit (1020) or input / output unit (1040), and audio stored in memory (120). The audio output unit (1070) may include at least one of a speaker (1071), a headphone output terminal (1072), or an S / PDIF (Sony / Philips Digital Interface: output terminal) (1073).

[0247] The user interface (1080) may receive user input for controlling the electronic device (100). The user interface (1080) may include, but is not limited to, various forms of user input devices, such as a touch panel that detects a user's touch, a button that receives a user's push operation, a wheel that receives a user's rotation operation, a keyboard, a dome switch, a microphone for voice recognition, and a motion detection sensor that senses motion. Additionally, when the electronic device (100) is operated by a remote controller (not shown), the user interface (1080) may receive a control signal received from the remote controller.

[0248] Hereinafter, with reference to FIGS. 11 to 14, it will be explained that the electronic device (100) of FIGS. 7 to 10 according to an embodiment outputs frames at regular intervals.

[0249] FIG. 11 is a diagram illustrating that, according to an embodiment, when the FPS of the input stream changes, the decoder (133) adjusts the number of frame repetitions and transmits them to the frame rate converter (134), thereby allowing frames to be output from the frame rate converter (134) at regular intervals.

[0250] Figure 11 assumes the case where the original FPS of the input stream changes from 30 FPS to 60 FPS.

[0251] In Table 11 (111), the decoder frame output time refers to the time when the decoder frame is output / acquired, i.e., the reconstructed image output from the decoder (133). The decoder frame output time may correspond to the input frame time of Tables 1 (10) to 5 (50) mentioned above. The input frame time indicates the time when a frame is input and stored in the frame buffer (920) within the frame rate converter ((135), and this may correspond to the time when the decoder output image is output / acquired from the decoder (133).

[0252] In an embodiment, it is assumed that the decoder (133) acquires a reconstructed image having a fixed FPS. In an embodiment, it is assumed that the fixed FPS of the reconstructed image acquired by the decoder (133) is 60 FPS.

[0253] In an embodiment, when the FPS of the input stream changes, the decoder (133) can obtain a restored image by adjusting the repetition count of the frame of the output image output from the decoder (133), thereby allowing the restored image with the adjusted frame repetition count to be input to the frame rate conversion unit (134). To this end, in an embodiment, the decoder (133) can adjust the number of restored frames obtained at the time of decoder frame output.

[0254] In an embodiment, it is assumed that while the electronic device (100) is playing the same content, the original FPS of the input stream changes from 30 FPS to 60 FPS. For example, as shown in Table 11 (111), it is assumed that the original FPS is 30 FPS until the restored image acquired by the decoder (133) is the 4th frame, and the original FPS changes to 60 FPS from the 6th frame onwards.

[0255] In the present disclosure, the time point represented by the frame is an integer based on a fixed FPS, and the time points corresponding to 0 seconds, 1 / 60 seconds, 2 / 60 seconds, 3 / 60 seconds, … are expressed as 0, 1, 2, 3, … and thus, in the interval where the original FPS is 60, the time point of the decoder frame output, that is, the output interval of the decoder frame output from the decoder (133), increases by 1, and in the interval where the original FPS is 30, the time point of the decoder frame output becomes 2.

[0256] In an embodiment, the decoder (133) can obtain original FPS information of the input stream from the header of the input stream.

[0257] In the embodiment, the original FPS information is information representing the frame rate of the video and can represent the number of image frames displayed on the screen per second.

[0258] In an example, the decoder (133) can identify that the FPS of the input stream has changed from 30 FPS to 60 FPS based on the original FPS information of the input stream.

[0259] In an embodiment, the decoder (133) can identify a changed frame acquisition pattern corresponding to a change in the FPS of the input stream.

[0260] In an embodiment, the memory (120) may have a change frame acquisition pattern stored in advance that is to be used to increase or decrease the time delay difference between input / output frames due to the change in the original FPS.

[0261] In an embodiment, the memory (120) may store a corresponding change frame acquisition pattern in the form of a matrix or a mapping table for each original FPS of the input stream before change and the original FPS of the input stream after change.

[0262] Since the original FPS of most broadcast or OTT content is set to standardized frame rates such as 24, 30, and 60, cases where the original FPS changes are limited, for example, when the original FPS changes from 24 to 30, from 24 to 60, or from 60 to 30.

[0263] In an embodiment, the decoder (133) can identify a change frame acquisition pattern corresponding to the original FPS of the input stream before change and the original FPS of the input stream after change from a matrix or mapping table stored in memory (120).

[0264] In an embodiment, the decoder (133) can identify a change frame acquisition pattern stored in response to the change in the input stream's FPS from 30 FPS to 60 FPS from a matrix or mapping table stored in memory (120).

[0265] In the embodiment, the modified frame acquisition pattern may be a pattern for reducing frame repetition by the amount of the increased time delay difference when the time delay difference between input and output frames increases due to the change in the original FPS, that is, when the time delay between the frame input to the frame rate converter (134) and the frame output from the frame rate converter (134) increases. That is, the modified frame acquisition pattern used when the original FPS decreases may be a pattern that offsets the increase in the time delay between input and output frames due to the decrease in the original frame rate.

[0266] In the embodiment, the modified frame acquisition pattern may be a pattern for increasing the number of repetitions of the same frame by the amount by which the time delay difference between input and output frames is reduced when the original FPS is changed. That is, the modified frame acquisition pattern used when the original FPS increases may be a pattern that offsets the reduction in time delay between input and output frames due to the increase in the original frame rate.

[0267] For example, if the original FPS of the input stream before change is 24 FPS and the original FPS of the input stream after change is 60 FPS, the corresponding change frame acquisition pattern may be a pattern that outputs the restored image obtained by decoding from the input stream two more times at the time when the original FPS is changed.

[0268] For example, if the original FPS of the input stream before change is 30 FPS and the original FPS of the input stream after change is 60 FPS, the corresponding change frame acquisition pattern may be a pattern that outputs the restored image obtained by decoding from the input stream one more time at the time when the original FPS is changed.

[0269] In an embodiment, if the original FPS of the input stream after change is higher than the original FPS of the input stream before change, the decoder (133) can add repeat frames instantaneously compared to a normal pull-down operation using a change frame acquisition pattern.

[0270] In the embodiment, the decoder (133) can acquire the restored image 6 once more at the time when the original FPS is changed according to the changed frame acquisition pattern, thereby allowing the 6th frame to be acquired twice.

[0271] That is, as indicated by drawing symbol 112 in Table 11 (111), when the original FPS of the input stream before change is 30 FPS and the original FPS of the input stream after change is 60 FPS, the decoder (133) can output video 6 twice as the decoder output video for a predetermined time from the time when the original FPS is changed to 60 FPS, that is, for the time during which two frames are restored according to the changed frame acquisition pattern.

[0272] In the section where the original FPS is 60, the time interval for outputting decoder frames, that is, the output interval of decoder frames output from the decoder (133), must increase by 1. Therefore, from the point where the original FPS changes to 60 in Table 11 (111), only one frame must be output. Accordingly, according to the existing technology, the 6th frame is output once, and then the 7th frame must be output.

[0273] That is, according to the prior art, as indicated in Table 4 (40) of FIG. 4 above, when the original FPS of the image changes from 30 FPS to 60 FPS, the decoder (133) must acquire / output the 6th frame once at the time when the original FPS changes to 60 FPS, and then acquire the next frame, the 7th frame, at a time interval of 1.

[0274] However, according to the embodiment, when the FPS of the input stream changes, the decoder (133) can compensate for the phenomenon where the difference between the input and output times becomes shorter by using a change frame acquisition pattern obtained based on the FPS of the input stream before the change and the FPS of the input stream after the change for the frame obtained at the time when the original FPS changed. That is, after the original FPS of the input stream changes to 60, the decoder (133) pulls down the 6th frame once more and outputs it, so the input frame is delayed by one frame and input to the frame rate conversion unit (134).

[0275] After the decoder (133) outputs the restored image 6 twice according to the changing frame acquisition pattern, it can acquire the restored image according to the frame acquisition pattern corresponding to the input stream. The frame acquisition pattern corresponding to the input stream can be determined according to the FPS and fixed FPS of the input stream.

[0276] In FIG. 11, since the original FPS of the changed input stream is 60 and is equal to the fixed FPS of the decoder (133), the decoder (133) does not perform a pull-down operation and can restore decoded frames 7, 8, etc. in order at intervals of 1.

[0277] In an embodiment, the decoder (133) can transmit the original FPS information of the frame and input stream obtained at the time of decoder frame output to the frame rate converter (134).

[0278] In an embodiment, the frame rate converter (134) receives frame output and original FPS information from the decoder (133) at the time of decoder frame output, and can extract only the key frame from the frame received from the decoder (133) and store it in the frame buffer (920).

[0279] Referring to Table 11 (111), the frame rate conversion unit (134) can obtain two keyframes from the frame buffer (920) for frame interpolation and perform frame interpolation.

[0280] In an embodiment, the frame buffer (920) may include three separate buffers. The first of the three buffers may be a buffer in which the keyframe currently acquired by the keyframe extraction unit (910) is stored. When the keyframe extraction unit (910) stores a newly acquired keyframe in the first buffer, the keyframe previously stored in the first buffer may be moved to the second frame buffer. In this case, the keyframe stored in the second frame buffer is moved to the third buffer. That is, data on the buffer is continuously changed in order for a new keyframe to be stored in the buffer. However, the same effect as changing the data may be obtained by changing the memory address pointed to by each frame buffer without actually changing the data on the buffer.

[0281] Here, among the keyframes used for frame interpolation, the most recent keyframe that was stored in the first buffer and moved to the second buffer is referred to as the second keyframe, and the keyframe that was stored in the second buffer prior to the second keyframe and moved to the third buffer upon the input of the second keyframe is referred to as the first keyframe.

[0282] In an embodiment, the frame rate conversion unit (134) can load the first key frame and the second key frame, respectively, and perform frame interpolation.

[0283] In the embodiment, when the original FPS is 30, the interpolation rate for frame rate conversion is performed at intervals of 0.25, and from the 6th frame, when the original FPS is changed to 60, which is the 1st keyframe, the interpolation rate for frame rate conversion is switched at intervals of 0.5 to match the changed FPS.

[0284] In the embodiment, the decoder (133) outputs the restored frame once more at the point where the original FPS changes to 60 and transmits it to the frame rate converter (134), so the frame rate converter (134) receives the input frame delayed by one frame from the decoder (133).

[0285] In an embodiment, the frame rate conversion unit (134) can output two frames at times 4.5 and 5 respectively by using frame 6 as the second key frame and frame 4 as the first key frame at the time when the decoder (133) restores frame 6 one more time, and interpolating the two frames by applying interpolation ratios of 0.25 and 0.5 to the two frames.

[0286] Afterwards, when the decoder (133) restores the 7th frame, the frame rate conversion unit (134) uses the 6th frame as the 2nd key frame, uses the 4th frame as the 1st key frame, and interpolates the two frames by applying interpolation ratios of 0.75 and 1, thereby outputting two frames at the time of 5.5 and 6 respectively.

[0287] That is, according to the present disclosure, even if the original FPS changes, it can be seen that the frame rate conversion unit (134) outputs the image at regular output times. This means that the difference in delay due to the difference in input / output times is constant regardless of whether the original FPS changes.

[0288] Unlike the part indicated by reference numeral 41 in Table 4 (40), where the output time increases by 2 from 4.5 to 6.5 in just two frames, according to Table 11 (111), as shown in the part indicated by reference numeral 114, it can be seen that the output time of the frame increases at a constant rate of 0.5.

[0289] Graph 11 (115) is a graph that compares the frame delay between the decoder frame output time and the frame rate converter frame output time of Table 11 (111). In Graph 11 (115), the x-axis represents time, and the y-axis represents the frame number or frame time. According to Graph 11 (115), it can be seen that the frame rate converter frame output time is maintained at a constant speed regardless of changes in the original FPS.

[0290] That is, when the original FPS changes from 30 FPS to 60 FPS, the time delay between input and output frames becomes shorter, but according to the present disclosure, when the original FPS changes, the decoder (133) performs a pull-down and outputs the frame one more time, thereby maintaining the time delay between input and output frames constant.

[0291] FIG. 11 illustrates a case where the decoder (133) adjusts the number of frames when the original FPS changes, but it is not limited thereto. As previously described in the description of FIG. 8, the encoder (800) may perform a pull-down to have a fixed FPS when generating the input stream. In this case, the decoder frame output time of Table 11 (111) can be interpreted as the time when the input stream / frame is acquired by the encoder (800).

[0292] In this case, the decoder (133) receives an input stream having a fixed FPS from the encoder (800) and can restore the decoder frame without a pull-down operation. The decoder (133) can transmit the original FPS information and the decoder frame to the frame rate converter (134). The frame rate converter (134) receives the decoder frame and the original FPS information from the decoder (133), performs frame interpolation, and outputs a frame rate converter frame in which the time delay difference between the input and output frames is constant.

[0293] FIG. 12 is a diagram illustrating that, according to an embodiment, when the FPS of the input stream changes, the decoder (133) adjusts the number of frame repetitions so that frames are output from the frame rate converter (134) at regular intervals.

[0294] In Table 12 (121), it is assumed that the original FPS of the input stream changes from 60 FPS to 30 FPS.

[0295] In Table 12 (121), the time of outputting the decoder frame is the time when the decoder output image is output / acquired from the decoder (133), and can correspond to the time of input frames in Tables 1 (10) to 5 (50) mentioned above.

[0296] In the embodiment, the decoder (133) assumes the case where it acquires a restored image having a fixed FPS of 60 FPS.

[0297] In an embodiment, the decoder (133) can adjust the number of frame restorations obtained at the time of decoder frame output when the FPS of the input stream changes, thereby allowing the restored image with the adjusted number of frame repetitions to be input to the frame rate converter (134).

[0298] In an embodiment, it is assumed that while the electronic device (100) is playing the same content, the original FPS of the input stream changes from 60 FPS to 30 FPS. For example, as shown in Table 12 (12), it is assumed that the original FPS is 6 FPS until the reconstructed image obtained from the decoder (133) is the 2nd frame, and the original FPS changes to 30 FPS from the 3rd frame onwards.

[0299] In the input frame, the time interval between input frames, i.e., the decoder frame output time, increases by 1 in the interval where the original FPS is 60, and in the interval where the original FPS is 30, the decoder frame output time increases by 2.

[0300] According to the existing technology, from the 3rd frame of the restored image in Table 12 (121), the original FPS is changed to 30 FPS, and the viewpoint interval between frames is changed to 2, so the 3rd frame is output twice, and then the 5th frame is output twice.

[0301] However, according to an embodiment, the decoder (133) can adjust the number of repetitions based on the FPS of the input stream before the change and the FPS of the input stream after the change for the frame acquired at the time when the original FPS is changed when the FPS of the input stream is changed.

[0302] In an embodiment, the decoder (133) obtains FPS information of the input stream from the header of the input stream and can identify a changed frame acquisition pattern corresponding to the change in the FPS of the input stream.

[0303] In an embodiment, the decoder (133) can identify a stored change frame acquisition pattern corresponding to the case where the FPS of the input stream changes from 60 FPS to 30 FPS from a matrix or mapping table stored in memory (120).

[0304] In an embodiment, the decoder (133) can acquire a restored image by adjusting the number of frame repetitions according to the identified change frame acquisition pattern.

[0305] In the embodiment, the change frame acquisition pattern may be a pattern for reducing the pulldown by the amount of the increased delay when the delay difference between the input / output timing increases due to a change in the original FPS, and for performing a pulldown when the delay difference decreases to increase the number of frames.

[0306] In an embodiment, when the original FPS of the input stream after modification is lower than the original FPS of the input stream before modification, the modified frame acquisition pattern may be a pattern that offsets the increase in delay of the frame rate conversion unit due to the decrease in the original frame rate by instantaneously skipping the repeating frame compared to a normal pull-down operation.

[0307] For example, if the original FPS of the input stream before change is 60 FPS and the original FPS of the input stream after change is 24 FPS, the corresponding change frame acquisition pattern may be a pattern that skips two repeating frames among the restored images obtained by decoding from the input stream at the time when the original FPS is changed.

[0308] For example, if the original FPS of the input stream before change is 60 FPS and the original FPS of the input stream after change is 30 FPS, the corresponding change frame acquisition pattern may be a pattern that skips non-keyframe repeating frames among the restored images obtained by decoding from the input stream at the time when the original FPS is changed.

[0309] In the embodiment, the decoder (133) can acquire the restored image 3 only once according to the changed frame acquisition pattern at the time when the original FPS is changed.

[0310] That is, as indicated by drawing symbol 122 in Table 12 (121), when the original FPS of the input stream before change is 60 FPS and the original FPS of the input stream after change is 30 FPS, the decoder (133) can output frame 3 only once as a decoder frame at the time when the original FPS is changed to 30 FPS.

[0311] If not according to the present disclosure, normally the third frame would have to be output twice when the original FPS changes to 30, but in this case, the delay of the output image obtained from the frame rate conversion unit (134) becomes longer.

[0312] Accordingly, in the embodiment, the decoder (133) can compensate for the reduced delay by outputting the first frame of the video only once instead of twice when the original FPS of the input stream changes to 30. Therefore, according to the present disclosure, even if the original FPS changes, the difference in delay between the input and output timing is eliminated, so that the frame rate conversion unit (134) can output the video at regular output timings.

[0313] In the embodiment, since the decoder (133) skips one restored frame at the point where the original FPS changes to 30, the frame rate converter (134) receives frames one frame faster from the decoder (133). Accordingly, the frame rate converter (134) performs frame interpolation with the frames received one frame faster from the decoder (133) and outputs a frame rate converter frame, and the frame rate converter frame obtained at this time has a fixed output time.

[0314] Graph 12 (125) is a graph showing a comparison of frame delays between the decoder frame output time and the frame rate conversion unit frame output time of Table 12 (121). According to Graph 12 (125), when the original FPS changes from 60 FPS to 30 FPS, the time delay between input and output frames increases, but according to the present disclosure, when the original FPS changes, the decoder (133) skips and outputs the frame, so the time delay between input and output frames is maintained at a constant level.

[0315] FIG. 12 illustrates a case where the decoder (133) adjusts the number of frames when the original FPS changes, but it is not limited thereto. As previously described in the description of FIG. 8, the encoder (800) may perform pull-down to have a fixed FPS when generating an input stream. The frame rate converter (134) receives decoder frame and original FPS information from the decoder (133), performs frame interpolation, and outputs a frame rate converter frame in which the time delay difference between the input and output frames is constant.

[0316] FIG. 13 is a diagram illustrating that, according to an embodiment, when the FPS of the input stream changes, the decoder (133) adjusts the number of frame repetitions so that frames are output from the frame rate converter (134) at regular intervals.

[0317] In Table 13 (131), it is assumed that the original FPS of the input stream changes from 24 FPS to 60 FPS.

[0318] In Table 13 (131), the time of outputting the decoder frame refers to the time when the decoder output image is output / acquired from the decoder (133).

[0319] In the embodiment, the decoder (133) assumes the case where it acquires a restored image having a fixed FPS of 60 FPS.

[0320] While the decoder (133) is playing the same content, if the original FPS changes to 60 FPS starting from the 10th frame of the restored video as shown in Table 13 (131), the time interval between frames changes to 1. Therefore, according to the existing technology, the 10th frame must be output only once at the time the original FPS changes. In this case, the delay of the output video obtained from the frame rate conversion unit (134) is shortened.

[0321] According to an embodiment, the decoder (133) obtains FPS information of the input stream from the header of the input stream and, corresponding to the change in the FPS of the input stream, can identify a change frame acquisition pattern stored in the matrix or mapping table stored in memory (120) in correspondence with the case where the FPS of the input stream changes from 24 FPS to 60 FPS.

[0322] In an embodiment, the decoder (133) can acquire a restored image by adjusting the number of frame repetitions according to the identified change frame acquisition pattern.

[0323] In the embodiment, the change frame acquisition pattern may be a pattern for increasing the number of identical frames by performing a pull-down when the delay difference between the input / output timing decreases due to the change in the original FPS.

[0324] In an embodiment, if the original FPS of the input stream after change is higher than the original FPS of the input stream before change, the change frame acquisition pattern may be a pattern that instantaneously adds repeat frames compared to a normal pull-down operation.

[0325] In the embodiment, the decoder (133) can output the 10th frame three times by using a change frame acquisition pattern, and at the time when the original FPS of the input stream changes to 60, by pulling down the 10th frame, which is the first frame of the video, as indicated by reference numeral 132, and adding it twice. Accordingly, the input frame is delayed by the two added frames and input to the frame rate converter (134), and accordingly, the first key frame and the second key frame used by the frame rate converter (134) are also changed, so that the output timing of the frame rate converter frame output from the frame rate converter (134) maintains a constant interval.

[0326] Graph 13 (135) is a graph showing a comparison of frame delays between the decoder frame output time and the frame rate conversion unit frame output time of Table 13 (131). According to Graph 13 (135), it can be seen that when the original FPS changes from 24 FPS to 60 FPS, the delay between input and output frames remains constant.

[0327] As described above in the description of FIG. 8, the encoder (800) may perform pull-down to have a fixed FPS when generating an input stream, thereby encoding the input stream so that the 10th frame is included three times. The frame rate converter (134) receives decoder frame and original FPS information from the decoder (133), performs frame interpolation, and outputs a frame rate converter frame in which the time delay difference between the input and output frames is constant.

[0328] FIG. 14 is a diagram illustrating that, according to an embodiment, when the FPS of the input stream changes, the decoder (133) adjusts the number of frame repetitions so that frames are output from the frame rate converter (134) at regular intervals.

[0329] In the embodiment, it is assumed that the decoder (133) acquires a restored image having a fixed FPS of 60 FPS.

[0330] Figure 14 assumes the case where the original FPS of the input stream changes from 60 FPS to 24 FPS.

[0331] In the input frame interval where the original FPS is 60, the input time interval between input frames, that is, the decoder frame output time, increases by 1, and in the interval where the original FPS is 24, the decoder frame output time increases by 2.5.

[0332] In an embodiment, the decoder (133) can adjust the number of frame restorations obtained at the time of decoder frame output when the FPS of the input stream changes, thereby allowing the restored image with the adjusted number of frame repetitions to be input to the frame rate converter (134).

[0333] In an embodiment, when the original FPS of the input stream changes from 60 FPS to 24 FPS while the electronic device (100) is playing the same content,

[0334] According to the existing technology, the decoder (133) performs a 3:2 pulldown to change the 24 FPS video to the fixed FPS of the decoder (133), which is 60 FPS, starting from the decoder frame acquired when the original FPS is changed to 24 FPS. For example, in Table 14 (141), the decoder (133) outputs the restored 5th frame three times when the decoder frame output time interval changes to 2.5, and then outputs the restored 7.5th frame twice. However, according to the existing technology, the delay of the output video acquired from the frame rate conversion unit (134) becomes longer.

[0335] In an embodiment, the decoder (133) can identify a stored change frame acquisition pattern in response to the change in the input stream's FPS from 60 FPS to 24 FPS, and acquire a restored image by adjusting the number of frame repetitions according to the identified change frame acquisition pattern.

[0336] In an embodiment, when the original FPS of the input stream after modification is lower than the original FPS of the input stream before modification, the modified frame acquisition pattern may be a pattern that offsets the increase in delay of the frame rate conversion unit due to the decrease in the original frame rate by instantaneously skipping the repeating frame compared to a normal pull-down operation.

[0337] In the embodiment, the decoder (133) identifies a change frame acquisition pattern corresponding to the original FPS of the input stream before change is 60 FPS and the original FPS of the input stream after change is 24 FPS, and can skip two repeat frames that need to be restored according to the identified change frame acquisition pattern.

[0338] In the embodiment, the decoder (133) can skip two frames of the 5th frame according to the changed frame acquisition pattern at the time when the original FPS is changed to 24 FPS, as indicated by drawing reference numeral 142 in Table 14 (141), and output the 5th frame only once instead of three times.

[0339] Accordingly, only one 5th frame is input to the frame rate conversion unit (134), and accordingly, the first key frame and the second key frame used by the frame rate conversion unit (134) are also changed, so that the output timing of the frame rate conversion unit frame output from the frame rate conversion unit (134) is maintained at a constant interval.

[0340] Graph 14 (145) is a graph showing a comparison of frame delays between the decoder frame output time and the frame rate converter frame output time of Table 14 (141). When the original FPS changes from 60 FPS to 24 FPS, the delay time between input and output frames increases. However, according to the present disclosure, as shown in Graph 14 (145), the decoder (133) skips and outputs frames at the time when the original FPS changes, so that the frame rate converter frame output time is maintained at a constant speed regardless of the change in the original FPS.

[0341] As described above in the description of FIG. 8, the encoder (800) may perform pull-down to have a fixed FPS when generating an input stream, thereby encoding so that the input stream contains the 5th frame only once. The frame rate converter (134) receives decoder frame and original FPS information from the decoder (133), performs frame interpolation, and outputs a frame rate converter frame in which the time delay difference between the input and output frames is constant.

[0342] FIG. 15 is a flowchart illustrating the operation method of an electronic device (100) according to an embodiment.

[0343] Referring to FIG. 15, the electronic device (100) can determine whether an input stream having a changed FPS is input (step 1510).

[0344] In an embodiment, the electronic device (100) obtains FPS information of the input stream from the header of the input stream and can identify whether the FPS of the input stream changes based on the FPS information of the input stream.

[0345] In an embodiment, when an input stream having a changed FPS is not input, the electronic device (100) can acquire a restored image according to a frame acquisition pattern corresponding to the current input stream (step 1520).

[0346] In an embodiment, when the electronic device (100) determines that an input stream having a changed FPS is input, it can acquire a restored image by adjusting the number of frame repetitions based on the FPS of the input stream before the change and the FPS of the input stream after the change (step 1530).

[0347] In an embodiment, the electronic device (100) can identify a change frame acquisition pattern based on the FPS of the input stream before change and the FPS of the input stream after change, and acquire a restored image by adjusting the number of frame repetitions based on the identified change frame acquisition pattern.

[0348] The method of operation of the electronic device (100) according to some embodiments and the electronic device (100) may also be implemented in the form of a recording medium containing instructions executable by a computer, such as a program module executed by a computer.

[0349] A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include both computer storage media and communication media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. A communication medium typically includes computer-readable instructions, data structures, program modules, or other data such as modulated data signals such as carrier waves, or other transmission mechanisms, and includes any information transmission medium.

[0350] In addition, the electronic device and the method of operation according to the embodiment of the present disclosure described above may be implemented as a computer program product comprising a computer-readable recording medium / storage medium having a program recorded thereon for implementing the method of operation of the electronic device, wherein the method of operation of the electronic device includes the step of decoding an input stream to obtain a restored image and the step of converting the frame rate (Frames Per Second, FPS) of the decoded image, and the step of obtaining the image includes, when the frame rate of the input stream is changed, the step of obtaining the image by adjusting the number of frame repetitions based on a change frame acquisition pattern corresponding to the frame rate of the input stream before change and the frame rate of the input stream after change, and the method further includes the step of converting the frame rate of the image obtained by adjusting the number of frame repetitions, and the step of outputting the image with the converted frame rate.

[0351] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0352] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Claims

In an electronic device (100), A decoder (133) that decodes an input stream to obtain an image; A frame rate conversion unit (134) that converts the frame rate (Frames Per Second, FPS) of the above-decoded video; A display unit (140) that outputs an image; Memory (120) for storing one or more instructions; and It includes at least one processor (110) including a processing circuit, and When the one or more instructions are executed individually or collectively by the at least one processor, The above decoder, under the control of the above processor, When the frame rate of the input stream is changed, the number of frame repetitions is adjusted based on a changed frame acquisition pattern corresponding to the frame rate of the input stream before the change and the frame rate of the input stream after the change to acquire the image, and The above frame rate conversion unit converts the frame rate of the image obtained by adjusting the number of frame repetitions according to the control of the processor, and An electronic device in which the display unit outputs an image with a frame rate converted by the frame rate conversion unit under the control of the processor. In claim 1, when the one or more instructions are executed individually or collectively by the at least one processor, The above decoder is an electronic device that, under the control of the processor, obtains FPS information of the input stream from the header of the input stream and identifies that the frame rate of the input stream has changed based on the FPS information of the input stream. In claim 2, when the one or more instructions are executed individually or collectively by the at least one processor, The above decoder is under the control of the above processor An electronic device that decodes the input stream to obtain the image having a fixed frame rate, and transmits the FPS information of the image having the fixed frame rate and the input stream to the frame rate converter. In claim 3, when the one or more instructions are executed individually or collectively by the at least one processor, The above decoder is under the control of the above processor The image is acquired according to a frame acquisition pattern corresponding to the input stream, which is determined according to the frame rate of the input stream and the fixed frame rate, and An electronic device that acquires an image by adjusting the number of frame repetitions according to a changed frame acquisition pattern, which is different from the frame acquisition pattern, when the frame rate of the input stream is changed. In claim 4, when the one or more instructions are executed individually or collectively by the at least one processor, The above decoder is under the control of the above processor An electronic device that identifies a change frame acquisition pattern stored in correspondence with the frame rate of the input stream before change and the frame rate of the input stream after change from a matrix or mapping table stored in the memory. In claim 4 or 5, when the one or more instructions are executed individually or collectively by the at least one processor, The above decoder is under the control of the above processor An electronic device that controls the number of frame repetitions of an image by using a pull-down method selected according to the difference in frame rates between the input stream frame rate and the fixed frame rate as the frame acquisition pattern when the fixed frame rate is higher than the frame rate of the input stream. In any one of claims 4 to 6, when the one or more instructions are executed individually or collectively by the at least one processor, The above decoder is under the control of the above processor An electronic device that acquires an image by increasing the number of repetitions of a repeating frame according to a first change frame acquisition pattern when the frame rate of the input stream changes from a first input frame rate to a second input frame rate, and the second input frame rate is higher than the first input frame rate. In any one of claims 4 to 6, when the one or more instructions are executed individually or collectively by the at least one processor, The above decoder is under the control of the above processor An electronic device that acquires an image by skipping a repeating frame according to a second change frame acquisition pattern when the frame rate of the input stream changes from a first input frame rate to a second input frame rate, if the second input frame rate is lower than the first input frame rate. In any one of claims 3 to 8, when the one or more instructions are executed individually or collectively by the at least one processor, The above frame rate converter, according to the control of the processor Receive FPS information of the image and the input stream from the decoder, and An electronic device that adds interpolated frames to the image using FPS information of the input stream and the fixed frame rate. In the method of operating an electronic device, A step of acquiring an image by decoding an input stream; and It includes a step of converting the frame rate (Frames Per Second, FPS) of the above-decoded video, and The step of acquiring the above image includes, when the frame rate of the input stream is changed, the step of acquiring the above image by adjusting the number of frame repetitions based on a changed frame acquisition pattern corresponding to the frame rate of the input stream before the change and the frame rate of the input stream after the change. The above method includes the step of converting the frame rate of the image obtained by adjusting the number of frame repetitions; and A method of operating an electronic device, further comprising the step of outputting an image with the above-mentioned frame rate converted. In claim 10, the step of obtaining FPS information of the input stream from the header of the input stream; and A method of operating an electronic device, further comprising the step of identifying that the frame rate of the input stream has changed based on the FPS information of the input stream. A method of operation of an electronic device according to claim 10 or 11, wherein the step of acquiring the image comprises the step of decoding the input stream to acquire the image having a fixed frame rate. In claim 12, the step of acquiring the image A step of acquiring the image according to a frame acquisition pattern corresponding to the input stream, which is determined according to the frame rate of the input stream and the fixed frame rate; and A method of operation of an electronic device comprising the step of acquiring the image by adjusting the number of frame repetitions according to the changed frame acquisition pattern, which is different from the frame acquisition pattern, when the frame rate of the input stream is changed. A method of operation of an electronic device according to claim 13, further comprising the step of identifying, from a previously stored matrix or mapping table, a change frame acquisition pattern stored in correspondence with the frame rate of the input stream before change and the frame rate of the input stream after change. A computer-readable recording medium having a program recorded thereon for performing the method of any one of claims 10 to 14 on a computer.

Images