Display device for processing image including object of interest and method therefor

Abstract

This display device is configured to: detect an object of interest included in a first frame of an image; acquire, on the basis of the shape of the object of interest, motion information indicating a motion of the object of interest between the first frame and a second frame; and add a motion compensation frame between the first frame and the second frame on the basis of the motion information. In order to acquire the motion information, the display device is further configured to set a plurality of weights including a first weight associated with a region of interest corresponding to the object of interest and a second weight associated with a background region different from the region of interest, wherein the second weight is different from the first weight, the plurality of weights are used for selecting, from among a plurality of candidate search blocks of the second frame, a matching block that matches a reference block including the object of interest of the first frame, and the reference block and the matching block are used for acquiring the motion information.

Description

Display device and method for processing an image including an object of interest

[0001] The present disclosure relates to a display device and method for processing an image including an object of interest (object).

[0002] When watching a video, viewers can focus their attention on a specific object. For example, when watching a sports game, viewers can focus their attention on the movement of a specific object, such as a ball.

[0003] Accurately estimating the movement of an object of interest can help provide viewers with natural-looking video. For example, precise motion compensation can be performed through the accurate estimation of the object's movement between consecutive frames. This allows viewers to be presented with video featuring smoother movement of the object.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of this document. None of the above is to be claimed as prior art related to this document, nor can it be used to determine prior art.

[0005] According to one aspect of the present disclosure, a display device comprises: a display; at least one processor including a processing circuit; The device includes a memory comprising at least one storage medium for storing one or more instructions, wherein when the one or more instructions are executed by the at least one processor, the display device is further configured to: detect an object of interest included in a first frame of an image, acquire motion information indicating the movement of the object of interest between the first frame and the second frame based on the shape of the object of interest, and add a motion compensation frame between the first frame and the second frame based on the motion information, and to acquire the motion information, when the one or more instructions are executed by the at least one processor, the display device is further configured to: set a plurality of weights including a first weight associated with a region of interest corresponding to the object of interest and a second weight associated with a background region different from the region of interest, wherein the second weight is different from the second weight, and the plurality of weights are used to select a matching block that matches a reference block containing the object of interest of the first frame among a plurality of candidate search blocks of the second frame, and the reference block and the matching block may be used to acquire the motion information.

[0006] In order to set the different weights, when the one or more instructions are executed by the at least one processor, the display device may further cause: to set a first weight value to a plurality of first pixels included in the region of interest and to set a second weight value to a plurality of second pixels included in the background region, and the first weight value may be greater than the second weight value.

[0007] In order to obtain the above motion information, when the above one or more instructions are executed by the at least one processor, the display device may further be caused to: select the matching block using a block matching algorithm according to a specified similarity measurement scheme, and obtain the motion information based on the reference block and the matching block.

[0008] In order to select the above matching block, when the above one or more instructions are executed by the above at least one processor, the display device may further cause to measure the similarity between the reference block and each candidate search block of the plurality of candidate search blocks using the specified similarity measurement method based on the above first weight value and the above second weight value.

[0009] The aforementioned specified similarity measurement method corresponds to the SAD (sum of squared differences) method, and the SAD method can be performed using the following mathematical formula:

[0010] [Mathematical Formula]

[0011]

[0012] Here, Pre(i,j) may represent the pixel value corresponding to position (i,j) in the reference block of the first frame (denote), Cur(i,j) may represent the pixel value corresponding to position (i,j) in the candidate search block among the plurality of candidate search blocks, wgt(i,j) may represent the weight value of the pixel corresponding to position (i,j), m may represent the vertical size of the candidate search block, n may represent the horizontal size of the candidate search block, and k may represent the index of the candidate search block.

[0013] In order to identify the shape of the object of interest, when the one or more instructions are executed by the at least one processor, the display device may further be caused to acquire information about the shape of the object of interest using a learned artificial intelligence model, and the learned artificial intelligence model may be trained to output output data including information about the shape of the object of interest based on an input frame.

[0014] The above output data may further include interest object detection information, and the interest object detection information may include at least one of type information for the type of the interest object, subtype information for the subtype of the interest object, or reliability information for the reliability level associated with the interest object.

[0015] In order to identify the shape of the object of interest, when the one or more instructions are executed by the at least one processor, the display device may further cause the image to sub-sample at a specified rate so that the object of interest is included in one block.

[0016] The first frame may be continuous with the second frame, the motion information may correspond to a motion vector for the object of interest, and the motion compensation frame may be generated based on the motion vector.

[0017] The above motion compensation frame can be used to perform at least one of frame rate conversion, motion compensation interpolation, or motion judder cancellation.

[0018] According to one aspect of the present disclosure, a method for controlling a display device comprises: detecting an object of interest included in a first frame of an image; acquiring motion information indicating the movement of the object of interest between the first frame and the second frame based on the shape of the object of interest; and adding a motion compensation frame between the first frame and the second frame based on the motion information, wherein the operation of acquiring the motion information comprises: setting a plurality of weights including a first weight associated with a region of interest corresponding to the object of interest and a second weight associated with a background region different from the region of interest, wherein the second weight is different from the second weight, and the plurality of weights are used to select a matching block that matches a reference block including the object of interest of the first frame among a plurality of candidate search blocks of the second frame, and the reference block and the matching block may be used to acquire the motion information.

[0019] The operation of setting different weights may include: setting a first weight value to a plurality of first pixels included in the region of interest and setting a second weight value to a plurality of second pixels included in the background region, and the first weight value may be greater than the second weight value.

[0020] The operation of acquiring the above motion information may include: an operation of selecting the matching block using a block matching algorithm according to a specified similarity measurement method; and an operation of acquiring the motion information based on the reference block and the matching block.

[0021] The operation of identifying the matching block may include: measuring the similarity between the reference block and each candidate search block of the plurality of candidate search blocks using the specified similarity measurement method based on the first weight value and the second weight value.

[0022] The aforementioned specified similarity measurement method corresponds to the SAD (sum of squared differences) method, and the SAD method can be performed using the following mathematical formula:

[0023] [Mathematical Formula]

[0024]

[0025] Here, Pre(i,j) may represent a pixel value corresponding to position (i,j) in the reference block of the first frame, Cur(i,j) may represent a pixel value corresponding to position (i,j) in the candidate search block among the plurality of candidate search blocks, wgt(i,j) may represent a weight value of the pixel corresponding to position (i,j), m may represent the vertical size of the candidate search block, n may represent the horizontal size of the candidate search block, and k may represent the index of the candidate search block.

[0026] The operation of identifying the shape of the object of interest may include: an operation of obtaining information about the shape of the object of interest using a learned artificial intelligence model, and the learned artificial intelligence model may be trained to output output data including information about the shape of the object of interest based on an input frame.

[0027] The above output data may further include interest object detection information, and the interest object detection information may include at least one of type information for the type of the interest object, subtype information for the subtype of the interest object, or reliability information for the reliability level associated with the interest object.

[0028] The operation of identifying the shape of the object of interest may include: an operation of sub-sampling the image at a specified ratio so that the object of interest is included in one block.

[0029] The first frame may be continuous with the second frame, the motion information may correspond to a motion vector for the object of interest, and the motion compensation frame may be generated based on the motion vector.

[0030] The above motion compensation frame can be used to perform at least one of frame rate conversion, motion compensation interpolation, or motion judder cancellation.

[0031] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken together with the accompanying drawings.

[0032] FIG. 1 is a configuration diagram of a display device according to one embodiment of the present disclosure.

[0033] FIG. 2 is a drawing illustrating an image including an object of interest according to one embodiment of the present disclosure.

[0034] FIG. 3 is a diagram illustrating the configuration of a display device for performing an operation to estimate the movement of an object of interest according to one embodiment of the present disclosure.

[0035] FIG. 4a is a drawing illustrating the movement of an object of interest in a series of frames including the object of interest, according to an embodiment of the present disclosure.

[0036] FIG. 4b is a drawing illustrating a region of interest and a background region corresponding to an object of interest according to one embodiment of the present disclosure.

[0037] FIG. 5 is a flowchart illustrating the operation of a display device estimating the movement of an object of interest according to one embodiment of the present disclosure.

[0038] FIGS. 6a to 6c are drawings illustrating the configuration of a display device for performing an operation to estimate the movement of an object of interest according to one embodiment of the present disclosure.

[0039] FIG. 7a is a drawing illustrating an artificial intelligence model used to detect an object of interest and / or the shape of an object of interest according to one embodiment of the present disclosure.

[0040] FIG. 7b is a drawing illustrating output data obtained through the artificial intelligence model of FIG. 7a according to one embodiment of the present disclosure.

[0041] FIG. 8 is a diagram illustrating the operation of a display device setting different weights for a region of interest and a background region corresponding to an object of interest, according to one embodiment of the present disclosure.

[0042] FIG. 9 is a diagram illustrating the operation of a display device subsampling an image including an object of interest according to one embodiment of the present disclosure.

[0043] FIG. 10 is a diagram illustrating the operation of a display device performing motion compensation based on motion information according to one embodiment of the present disclosure.

[0044] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0045] According to the embodiments, identical or similar reference numerals may be used to refer to identical or similar elements throughout the present disclosure.

[0046] FIG. 1 is a configuration diagram of a display device according to one embodiment of the present disclosure.

[0047] According to one embodiment, the display device (100) may be a smartphone, a personal computer (PC), a tablet PC, a smart TV, a mobile phone, a PDA (personal digital assistant), a laptop, a media player, a micro server, a digital broadcasting terminal, a navigation device, a kiosk, a home appliance, and other mobile or non-mobile computing devices, but is not limited thereto. The display device (100) may perform various computing functions such as real-time video viewing and communication. In the following description, the display device (100) is described on the premise that it is a TV or a monitor, but this is merely an example and the embodiments of the present disclosure may be equally applied to electronic devices having display functions.

[0048] Referring to FIG. 1, the display device (100) may include at least one processor (110), memory (120), image input unit (130), display (140), and communication unit (150).

[0049] According to one embodiment, memory (120) is a storage medium used by a display device (100) and can store data such as one or more instructions (121) (e.g., one or more commands) or configuration information corresponding to at least one program. The program may include an operating system (OS) program and various application programs. When the instructions (121) stored in memory (120) are executed on at least one processor (110), they can cause the display device (100) to perform at least one operation (e.g., at least one of the operations described later with reference to FIGS. 2 to 10).

[0050] According to one embodiment, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD (secure digital) or XD (extreme digital) memory), RAM (random access memory), SRAM (static random access memory), ROM (read only memory), EEPROM (electrically erasable programmable ROM), PROM (programmable ROM), magnetic memory, a magnetic disk, or an optical disk.

[0051] According to one embodiment, the video input unit (130) may receive video data through (e.g., by using) at least one of a tuner, an input / output unit, or a communication unit (150). The video input unit (130) may include at least one of the tuner and the input / output unit. The tuner may select only the frequency of the broadcast channel to be received by the display device (100) from among many radio wave components by tuning through processing such as amplification, mixing, and resonance of a broadcast signal received via wired or wireless means. The broadcast signal may include video, audio, and additional data (e.g., an EPG (electronic program guide)). The tuner may receive broadcast channels (or viewing video) from various broadcast sources such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and internet broadcasting. The tuner may be implemented as an integral unit with the display device (100) or as a separate tuner electrically connected to the display device (100). The above input / output unit may include at least one of an HDMI (high definition multimedia interface) input port, a component input jack, a PC input port, or a USB input jack, which can receive video data from an external device of the display device (100) under the control of the processor (110). It is obvious to those skilled in the art that the above input / output unit may be added, deleted, and / or changed depending on the performance and structure of the display device (100).

[0052] According to one embodiment, the display (140) can perform functions for outputting information in the form of numbers, characters, images, and / or graphics. The display (140) may include at least one hardware module for output. The at least one hardware module may include, for example, at least one of an LCD (liquid crystal display), an LED (light emitting diode), an LPD (light emitting polymer display), an OLED (organic light emitting diode), an AMOLED (active matrix organic light emitting diode), or an FLED (flexible LED). The display (140) can display a screen corresponding to data received from the processor (110). The display (140) may be referred to as an 'output unit', a 'display unit', or other terms having an equivalent technical meaning.

[0053] According to one embodiment, the communication unit (150) includes a communication circuit and may provide a wired / wireless communication interface that enables communication with an external device. The communication unit (150) may include at least one of a wired Ethernet, a wireless LAN (local area network) communication unit, and a local area communication unit. The wireless LAN communication unit may include, for example, Wi-Fi and may support the wireless LAN standard (IEEE802.11x) of the Institute of Electrical and Electronics Engineers (IEEE). The wireless LAN communication unit may be wirelessly connected to an access point (AP) under the control of the processor (110). The local area communication unit may wirelessly perform local area communication with an external device under the control of the processor (110). Near-field communication may include Bluetooth, Bluetooth Low Energy (BLE), infrared data association (IrDA), ultra-wideband (UWB), WiFi Direct, and / or near-field communication (NFC). The external device may include a server device that provides services such as video services and a mobile terminal (e.g., phone, tablet, etc.).

[0054] According to one embodiment, at least one processor (110) can perform operations or data processing regarding the control and / or communication of at least one other component of the display device (100) by executing at least one instruction (121) stored in memory (120). The processor (110) may include at least one processing circuit that executes instructions stored in memory (120).

[0055] According to one embodiment, at least one processor (110) may include various processing circuits and / or multiple processors. One or more of the at least one processor (110) may be configured to perform various functions described in this disclosure individually and / or collectively. As used herein, where "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms may cover, for example, a situation in which one processor performs some of the cited functions and other processor(s) perform other parts of the cited functions, and may also cover, but are not limited to, a situation in which a single processor can perform all of the cited functions. Additionally, at least one processor (110) may include a combination of processors performing the cited / disclosed various functions, for example, in a distributed manner. At least one processor (110) may execute program instructions to achieve or perform various functions.

[0056] According to one embodiment, at least one processor (110) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have a plurality of cores.

[0057] According to one embodiment, at least one processor (110) can, for example, execute software to control at least one other component (e.g., a hardware or software component) of a display device (100) connected to the processor (110) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (110) can store instructions or data received from other components in volatile memory, process instructions or data stored in volatile memory, and store result data in non-volatile memory. According to one embodiment, the processor (110) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit, an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with it. For example, if the display device (100) includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use less power than the main processor or to perform a designated function. The auxiliary processor can be implemented separately from the main processor or as part of it.

[0058] According to one embodiment, the processor (110) may obtain video frame data from at least one of memory (120), video input unit (130), or communication unit (150). The processor (110) may receive video frame data from at least one of memory (120), display (140), video input unit (130), or communication unit (150). The video frame data may include data regarding a frame included in the video. According to embodiments, the video may be a video, and the frame may be a frame included in the video. For example, the video frame data may be identified from memory (120) (e.g., a video that was previously recorded and stored). For example, the video frame data may include data obtained from the communication unit (150) or video input unit (130) (e.g., a real-time streaming video).

[0059] FIG. 2 is a drawing illustrating an image including an object of interest according to one embodiment of the present disclosure.

[0060] Referring to FIG. 2, the image (200) (or, a frame of the image) may include at least one object of interest (211). According to embodiments, when the object of interest (211) is described as being included in the image (200) or a frame of the image (200), it may mean that an image portion corresponding to the object of interest (211) is included in the image (200) or a frame of the image (200). For example, this may mean that the representation and reflection of the object of interest is included in the image (200) or a frame of the image (200). According to embodiments, the frame may be referred to as an image frame.

[0061] According to one embodiment, the object of interest (211) may be an object that is the object of interest of a viewer. For example, the object of interest (211) may be an object that a viewer watching the video (200) takes an interest in within the video (200). The object of interest (211) may be associated with the content of the video (200). For example, as illustrated in FIG. 2, based on the fact that the video (200) is a video having content of a sports game such as a baseball game, a viewer of the video (e.g., a viewer of the video (200)) may focus their attention on the movement of the ball used in the sports game. As a result, the object of interest (211) may be a ball used in the sports game (e.g., a baseball).

[0062] According to one embodiment, a display device (e.g., the display device (100) of FIG. 1) can set an object of interest (211) of an image (200) based on the content of the image (200). For example, the display device (100) can analyze the content of the image (200) (e.g., analyze the type of content) and, based on the analysis result (e.g., based on the result of the analysis), set at least one of the objects included in the image (200) as an object of interest (211).

[0063] According to one embodiment, a display device (e.g., the display device (100) of FIG. 1) may set an object of interest (211) of an image (200) based on a specified setting. The specified setting may include, for example, a setting based on user input, signaling information included in the image (200), or a setting based on metadata of the image (200), but the embodiments are not limited thereto. The signaling information included in the image (200) may include information about an object of interest (211) of the image (200), provided together with or separately from the image (200) by a device (e.g., a content provider) that transmitted the image (200) to the display device (100). Information about the object of interest (211) may include, for example, the type of the object of interest (211) (e.g., a ball), the subtype of the object of interest (211) (e.g., a baseball), the shape of the object of interest (211) (e.g., circular, elliptical), the size of the object of interest (211), and / or other information related to the object of interest (211).

[0064] According to one embodiment, an object of interest (211) may be included in at least one block (210) within a frame of an image (200). For example, as illustrated in FIG. 2, a frame of an image (200) may be partitioned into blocks having a specified size (e.g., 8*8 or 16*16 size), and an object of interest (211) may be included in one block (210). Meanwhile, based on the object of interest (211) being included in a plurality of blocks, for accurate motion estimation of the object of interest (211), the display device (100) may process (e.g., subsample) the object of interest (211) so that it is included in one block (210). An example of the operation of subsampling the image (200) so that an object of interest (211) included in a plurality of blocks is included in one block (210) is described below with reference to FIG. 9.

[0065] Hereinafter, with reference to FIG. 3, an example of an operation for estimating the movement of an object of interest (211) will be described. For convenience of explanation, the following description will use the example of an object of interest (211) being a public object, but the embodiments are not limited thereto. For example, various types of objects of interest (211) having movement characteristics (e.g., a racing car in a racing broadcast video) may be applied to various embodiments of the present disclosure.

[0066] FIG. 3 is a diagram illustrating the configuration of a display device for performing an operation to estimate the movement of an object of interest according to one embodiment of the present disclosure.

[0067] FIG. 4a is a drawing illustrating the movement of an object of interest in a series of frames including the object of interest, according to an embodiment of the present disclosure.

[0068] FIG. 4b is a drawing illustrating a region of interest and a background region corresponding to an object of interest according to one embodiment of the present disclosure.

[0069] Referring to FIGS. 3 to 4b, a display device (300) (e.g., may correspond to the display device (100) of FIG. 1) may include an object detection unit (310), a motion estimation unit (320), and / or a motion interpolation unit (330). Depending on the embodiment, some of the above-described components (e.g., the motion interpolation unit (330)) may be omitted, and additional components may be included.

[0070] According to one embodiment, the object detection unit (310), the motion estimation unit (320), and / or the motion interpolation unit (330) may be implemented using at least one processor (e.g., the processor (110) of FIG. 1). For example, instructions may be stored in memory (e.g., the memory (120) of FIG. 1), and when these instructions are executed by at least one processor (110), they may cause the display device (300) to perform at least one function and / or operation of the object detection unit (310), the motion estimation unit (320), and / or the motion interpolation unit (330).

[0071] According to one embodiment, the object detection unit (310) receives input data (301) including at least one frame and can detect an object of interest (e.g., an object of interest (211) of FIG. 2) included in each frame using a specified object detection algorithm. For example, as illustrated in FIG. 4a, the object detection unit (310) can receive a first frame (exemplified as "N-1 frame") and detect an object of interest (401) (e.g., a baseball) included in a first block (410) of the first frame. For example, as illustrated in FIG. 4a, the object detection unit (310) can receive a second frame (exemplified as "N frame") and detect an object of interest (401) included in a second block (420) of the second frame. The first frame and the second frame may be consecutive frames. The object detection unit (310) can transmit data of the detected object of interest (401) to the motion estimation unit (320). The data of the object of interest (401) may include, for example, location information of the object of interest (401). The location information of the object of interest (401) may include, for example, the location coordinates of the block (410) and the block (420) containing the object of interest (401) within the frame containing the object of interest (401).

[0072] According to one embodiment, the motion estimation unit (320) receives data (e.g., location information of the object of interest (401)) from the object detection unit (310) and can obtain motion information (e.g., motion vector) by estimating the motion of the object of interest (401) between consecutive frames. The motion estimation unit (320) can obtain motion information for the object of interest (401) by using a fixed block matching algorithm. The fixed block matching algorithm may be a block matching algorithm that always uses blocks of a fixed size and shape without considering the shape of the object of interest (401). For example, as illustrated in FIG. 4a, based on the fact that the object of interest (401) moves from the position of the first block (410) to the position of the second block (420) between the first frame and the second frame, the motion estimation unit (320) can determine (e.g., select) the second block (420) as the matching block of the first block (410) using a fixed block matching algorithm, and obtain a motion vector for the object of interest (401) based on the first block (410) and the second block (420).

[0073] According to one embodiment, the motion interpolation unit (330) can interpolate an image based on motion information. For example, the motion interpolation unit (330) can interpolate an image by performing motion compensation based on motion information.

[0074] As described above, the motion estimation unit (320) of FIG. 3 performs motion estimation for the object of interest (401) using a fixed block matching algorithm that always uses blocks of a fixed size and shape without considering the shape of the object of interest (401), so it may be difficult to accurately estimate motion in a specific environment or situation. For example, in the example of the block (410) and the example of the block (420) shown in FIG. 4b, it may be difficult to accurately estimate motion for the object of interest (401b) based on the fact that the complexity of the background area (402b-1) and background area (402b-2), which are not the area of ​​interest corresponding to the object of interest (401b), is relatively high, or the shape in which the object of interest (401b) is captured by the camera changes. Therefore, in order to accurately estimate the position of the object of interest (401b), a motion estimation method may be used to reduce the influence of the background area (402b-1, 402b-2) by considering the shape of the object of interest (401b).

[0075] Hereinafter, with reference to FIGS. 5 to 10, an example of a method for performing motion estimation considering the shape of an object of interest will be described.

[0076] FIG. 5 is a flowchart illustrating the operation of a display device estimating the movement of an object of interest according to one embodiment of the present disclosure.

[0077] FIGS. 6a to 6c are configuration diagrams illustrating the configuration of a display device for performing an operation to estimate the movement of an object of interest according to one embodiment of the present disclosure.

[0078] FIG. 7a is a drawing illustrating an artificial intelligence model used to detect an object of interest and / or the shape of an object of interest according to one embodiment of the present disclosure.

[0079] FIG. 7b is a drawing illustrating output data obtained through the artificial intelligence model of FIG. 7a according to one embodiment of the present disclosure.

[0080] FIG. 8 is a diagram illustrating the operation of a display device setting different weights for a region of interest and a background region corresponding to an object of interest, according to one embodiment of the present disclosure.

[0081] FIG. 9 is a diagram illustrating the operation of a display device subsampling an image including an object of interest according to one embodiment of the present disclosure.

[0082] FIG. 10 is a diagram illustrating the operation of a display device performing motion compensation based on motion information according to one embodiment of the present disclosure.

[0083] Referring to FIGS. 5 through 6c, a display device (e.g., at least one of a display device (600a), a display device (600b), and a display device (600c)) can obtain output data (602) by using input data (601) comprising at least one frame of an image (e.g., an image (200) of FIG. 2) to detect an object of interest (e.g., an object of interest (211) of FIG. 2) and / or the shape of the object of interest, to estimate motion based on the shape of the object of interest, and / or to perform motion compensation (or interpolation) based on the motion estimation. The output data (602) can be transmitted to at least one other configuration for image processing. According to embodiments, one or more of the display device (600a), the display device (600b), and the display device (600c) may correspond, for example, to the display device (100) of FIG. 1.

[0084] According to one embodiment, each of the display devices (600a, 600b, and 600c) may include an object detection unit (610), an object shape detection unit (620), a motion estimation unit (630), and / or a motion interpolation unit (640). Depending on the embodiment, some of the above-described components (e.g., a motion interpolation unit (330)) may be omitted, and additional components may be included. Depending on the embodiment, a plurality of the above-described components may be integrated into a single component. For example, the object detection unit (610) and the object shape detection unit (620) may be integrated into a single component. In this case, the integrated component may perform the operation for detecting an object of interest in an image and the operation for detecting the shape of the object of interest together. For example, the integrated component may use a single trained artificial intelligence model to perform the operation for object detection for each frame of the image and the operation for detecting the shape of the object together.

[0085] According to one embodiment, the object detection unit (610), the object shape detection unit (620), the motion estimation unit (630), and / or the motion interpolation unit (640) may be implemented using at least one processor (e.g., the processor (110) of FIG. 1). For example, instructions may be stored in memory (e.g., the memory (120) of FIG. 1), and when these instructions are executed by at least one processor (110), they may cause the display device (100) to perform at least one function and / or operation of the object detection unit (610), the object shape detection unit (620), the motion estimation unit (630), and / or the motion interpolation unit (640).

[0086] According to one embodiment, as illustrated in FIG. 6a, the operation of the object detection unit (610) may be performed before the operation of the object shape detection unit (620), but the embodiments are not limited thereto. For example, as illustrated in FIG. 6b and 6c, the operation of the object detection unit (610) may be performed after the operation of the object shape detection unit (620), or, as described above, together.

[0087] According to one embodiment, as illustrated in FIG. 6c, the display device (600c) may further include a motion estimation unit (631) separate from the motion estimation unit (630) for the object of interest. The motion estimation unit (631) may, for example, estimate the motion of an object other than the object of interest in consecutive frames. In this case, the motion interpolation unit (640) can increase the accuracy of the motion compensation by performing motion compensation using both the motion estimation results of the motion estimation unit (630) and the motion estimation unit (631).

[0088] At least one operation of the object detection unit (610), object shape detection unit (620), motion estimation unit (630) and / or motion interpolation unit (640) described above may correspond to one or more of the operations exemplified in FIG. 5.

[0089] According to one embodiment, in operation 510 of FIG. 5, the display device (100) can identify or detect an object of interest (e.g., the object of interest (401) of FIG. 4a) (or the shape of the object of interest) included in the first frame (e.g., the N-1 frame of FIG. 4a). Operation 510 can be performed, for example, by an object detection unit (610) and / or an object shape detection unit (620) of FIG. 6a to 6c.

[0090] According to one embodiment, the operation of identifying the shape of an object of interest may be performed after or before the display device (100) detects the object of interest, or may be performed together with the detection of the object of interest. The operation of the display device (100) detecting the object of interest may be performed, for example, by the object detection unit (610) of FIGS. 6a to 6c.

[0091] In the following, first, an example of an operation in which a display device (100) detects (or identifies) an object of interest in an image is described. The object of interest may be detected, for example, using a learned artificial intelligence model and / or edge information of the image, but the embodiments are not limited thereto. In the following description, the operation of the display device (100) for detecting an object of interest may correspond to the operation of an object detection unit (610).

[0092] According to one embodiment, a display device (100) can detect an object of interest included in a frame using a learned first artificial intelligence model (e.g., the artificial intelligence model (700) of FIG. 7a). For example, as illustrated in FIG. 7a, the display device (100) can input a frame (701) into the learned first artificial intelligence model (e.g., the frame (701) can be provided as an input to the learned first artificial intelligence model) and obtain output data from the first artificial intelligence model that includes object of interest detection information for at least one object of interest included in the frame (e.g., the first ball (B1) and / or the second ball (B2)).

[0093] According to one embodiment, the object of interest detection information may include, for example, type information regarding the type of the object of interest, subtype information regarding the subtype of the object of interest, and / or reliability information regarding the reliability of the object of interest (e.g., at least one of reliability associated with or corresponding to the object of interest, reliability level, and reliability value). For example, the first object of interest detection information for the first ball (B1) of FIG. 7a may include type information indicating that the type of the object of interest is a ball (e.g., "Ball"), subtype information indicating that the subtype of the object of interest is a baseball (e.g., "Type1"), and / or reliability information indicating that the reliability of the object of interest (e.g., reliability level associated with the object of interest) is 95% (e.g., "probability of 0.95"), as illustrated in the example of block (BL6) illustrated in FIG. 7b. The display device (100) can identify (or determine) that a baseball is included in the frame with a 95% reliability level through the first object of interest information. For example, the second object of interest detection information for the second ball (B2) of FIG. 7a may include type information indicating that the type of the object of interest is a ball (e.g., "Ball"), subtype information indicating that the subtype of the object of interest is a rugby ball (e.g., "Type 2"), and / or confidence information indicating that the confidence level of the object of interest (e.g., confidence level associated with the object of interest) is 85% (e.g., "probability of 0.85"), as illustrated in the example of block (BL1) illustrated in FIG. 7b. The display device (100) can use the second object of interest information to identify that a rugby ball is included in the frame with an 85% confidence level. According to an embodiment, the type information and subtype information may be integrated into a single piece of information.When detecting an object of interest by obtaining object of interest detection information for the object of interest through the first artificial intelligence model trained in this way, the object of interest can be detected quickly and accurately compared to other approaches (e.g., approaches using edge information).

[0094] According to one embodiment, the first artificial intelligence model may be a convolutional neural network (CNN) model or a region-based CNN (R-CNN) model, but the embodiments are not limited thereto. The first artificial intelligence model may be trained using a specified learning scheme. The specified learning scheme may be, for example, a supervised learning scheme, an unsupervised learning scheme, or a reinforcement learning scheme. For example, based on the use of a supervised learning scheme, the first artificial intelligence model may be trained based on training data comprising frames containing at least one labeled object of interest. For example, a frame containing a baseball (e.g., the first ball (B1)) as the object of interest may be labeled with first data including a ball (type), a baseball (subtype), and / or a probability value 1 (e.g., may correspond to a confidence level of 100%).

[0095] According to one embodiment, the display device (100) can detect an object of interest included in a frame by using edge information. An edge may be, for example, a part where a pixel value (or brightness) changes abruptly in an image (or frame), and may be used as information representing the outline or boundary of an object.

[0096] According to one embodiment, a display device (100) can obtain edge information by detecting edges within a frame using a specified edge detection algorithm. The specified edge detection algorithm is an algorithm that detects edges based on brightness differences between pixels of a frame, and may be, for example, at least one of the Sobel algorithm, the Prewitt algorithm, or the Canny algorithm. The display device (100) can extract the boundary line of an object based on the edge information and use the extracted boundary line to confirm or determine whether an object of interest exists within the frame. For example, the display device (100) can determine whether an object of interest exists within the frame by determining whether the extracted boundary line matches a specified characteristic (e.g., pattern, size, shape) of the object of interest. Based on the determination that the object of interest exists within the frame, the display device (100) can obtain location information (e.g., location coordinates) of the object of interest within the frame. By using this process, the display device (100) can detect an object of interest included in the frame.

[0097] Hereinafter, an example of an operation in which a display device (100) detects (or identifies) the shape of an object of interest in an image is described. The shape of the object of interest may be detected, for example, using a learned artificial intelligence model and / or edge information of the image, but the embodiments are not limited thereto. In the following description, the operation of the display device (100) for detecting the shape of an object of interest may correspond to the operation of an object shape detection unit (620).

[0098] According to one embodiment, a display device (100) can detect (or identify) the shape of an object of interest included in a frame using a learned second artificial intelligence model (e.g., the artificial intelligence model (700) of FIG. 7a). For example, as illustrated in FIG. 7a, the display device (100) inputs a frame (701) into a learned second artificial intelligence model and obtains output data from the second artificial intelligence model that includes shape information of an object of interest for at least one object of interest (e.g., a first ball (B1) and / or a second ball (B2)).

[0099] According to one embodiment, the shape information of an object of interest may include information regarding the shape of the object of interest. The information regarding the shape of the object of interest may include, for example, information regarding an area corresponding to the shape of the object of interest (hereinafter, area of ​​interest). For example, the first shape information of an object of interest for the first ball (B1) of FIG. 7a may include information indicating that the area of ​​interest corresponding to the shape of the object of interest is a first area (e.g., "Area 1"), as illustrated in the example of block (BL6) illustrated in FIG. 7b. For example, the second shape information of an object of interest for the second ball (B2) of FIG. 7b may include information indicating that the area of ​​interest corresponding to the shape of the object of interest is a second area (e.g., "Area 2"), as illustrated in the example of block (BL1) illustrated in FIG. 7b. Through this, the display device (100) can identify the area of ​​interest occupied by the object of interest within a frame (or a specific block of the frame). The region of interest may be an area corresponding to, for example, a bounding box surrounding the object of interest (e.g., the first bounding box (BB1) and the second bounding box (BB2) in FIG. 7a). In this case, information regarding the region of interest corresponding to the shape of the object of interest may include location coordinates indicating the position of the bounding box within the frame. The setting of the shape of the bounding box (e.g., rectangle, polygon) may vary depending on the type of object of interest, the required latency, and the performance of the processor. For example, as the performance of the processor increases, the bounding box is set to fit the boundary of the object of interest more closely, so that the region of interest can be identified more accurately.

[0100] According to one embodiment, the second artificial intelligence model may be a CNN model or an R-CNN model, but the embodiments are not limited thereto. The second artificial intelligence model may be trained using a specified learning method. The specified learning method may be, for example, a supervised learning method, an unsupervised learning method, or a reinforcement learning method. For example, based on the use of a supervised learning method, the second artificial intelligence model may be trained based on training data comprising frames containing at least one labeled object of interest. For example, a frame containing a baseball (e.g., a first ball (B1)) as an object of interest may be labeled with second data containing an area corresponding to the baseball (e.g., "Area 1").

[0101] According to one embodiment, a display device (100) can detect the shape of an object of interest included in a frame using edge information. The display device (100) can extract the boundary lines of an object based on the edge information and generate a closed curve of the object by connecting consecutive boundary lines. The display device (100) can identify the shape of an object of interest using the generated closed curve. The process of detecting the shape of an object of interest using such edge information may have slightly lower performance compared to the process of detecting the shape of an object of interest using a learned artificial intelligence model, as it is affected by the complexity of the background, the complexity of the object's shape, and noise.

[0102] According to the embodiment, detection of an object of interest and detection of the shape of an object of interest may be performed together or simultaneously. For example, a display device (100) may use a single integrated artificial intelligence model (which may be referred to as a third artificial intelligence model) to perform detection of an object of interest and detection of the shape of an object of interest for a frame together. In this case, as illustrated in FIG. 7a, the display device (100) inputs a frame (701) into a learned third artificial intelligence model and obtains output data from the third artificial intelligence model that includes both object of interest detection information and object of interest shape information for at least one object of interest (e.g., first ball (B1) and / or second ball (B2)). For example, the first output data for the first ball (B1) of FIG. 7a may include, as illustrated in the example of block (BL6) in FIG. 7b, first object of interest detection information including type information indicating that the type of the object of interest is a ball, subtype information indicating that the subtype of the object of interest is a baseball, and / or reliability information indicating that the reliability of the object of interest is 95%, and first object of interest shape information indicating that the area of ​​interest corresponding to the shape of the object of interest is a first area (e.g., "Area 1"). Through this, the display device (100) can identify that a baseball is included in the first area with a 95% reliability level within the frame. For example, the second output data for the second ball (B2) of FIG. 7a may include, as illustrated in the example of block (BL1) in FIG. 7b, second object of interest detection information including type information indicating that the type of the object of interest is a ball, subtype information indicating that the subtype of the object of interest is a rugby ball, and / or reliability information indicating that the reliability of the object of interest is 85%, and first object of interest shape information indicating that the area of ​​interest corresponding to the shape of the object of interest is a second area (e.g., "Area 2").The display device (100) can identify that a rugby ball is contained in a second area within the frame with an 85% confidence level.

[0103] According to one embodiment, the third artificial intelligence model may be a CNN model or an R-CNN model, but the embodiments are not limited thereto. The third artificial intelligence model may be trained using a specified learning method. The specified learning method may be, for example, a supervised learning method, an unsupervised learning method, or a reinforcement learning method. For example, based on the use of a supervised learning method, the third artificial intelligence model may be trained based on training data comprising frames containing at least one labeled object of interest. For example, a frame containing a baseball (e.g., a first ball (B1)) as an object of interest may be labeled with third data including a ball (type), a baseball (subtype), an area corresponding to the baseball (Area 1), and / or a probability value 1 (e.g., may correspond to a confidence level of 100%).

[0104] In operation 520, the display device (100) can acquire motion information (e.g., motion vector) indicating the movement of the object of interest between a first frame (e.g., "N-1 frame" of FIG. 4a) and a second frame (e.g., "N frame" of FIG. 4a) based on the shape of the object of interest. The first frame and the second frame may be consecutive frames. For example, the second frame may be the frame immediately following the first frame. Operation 520 may be performed by, for example, the motion estimation unit (630) of FIG. 6a to 6c.

[0105] According to one embodiment, the operation of acquiring motion information of operation 520 may include an operation of setting different weights for a region of interest corresponding to the shape of an object of interest and a background region that is not a region of interest. For example, the operation of acquiring motion information of operation 520 may include an operation of setting a plurality of weights including a first weight associated with a region of interest corresponding to the shape of an object of interest and a second weight associated with a background region that is not a region of interest. For example, a display device (100) may set a first weight value for pixels included in the region of interest (e.g., a plurality of first pixels) and set a second weight value for pixels included in the background region (e.g., a plurality of second pixels). The first weight value may be greater than the second weight value. According to one embodiment, the region of interest and the background region may be included in a single block.

[0106] According to one embodiment, the operation of acquiring motion information of operation 520 may be performed using a block matching algorithm according to a specified similarity measurement method. The block matching algorithm may be, for example, an algorithm that determines the most similar block as the block that matches the reference block (which may be referred to as the matching block) by comparing a reference block (e.g., a block of interest) of a reference frame and search blocks (e.g., candidate search blocks) of a search frame using a specified similarity measurement method. For example, the operation of acquiring motion information of operation 520 may include the operation of identifying or selecting a matching block that matches the reference block (e.g., a block containing an object of interest of the first frame) among a plurality of candidate search blocks of the second frame using a block matching algorithm according to a specified similarity measurement method, and the operation of acquiring motion information for an object of interest based on the reference block and the matching block. The operation of identifying a matching block may include an operation of measuring the similarity between a reference block and each candidate search block among a plurality of candidate search blocks (e.g., measuring SAD values) using a set first weight value and a set second weight value, and using a specified similarity measurement method (e.g., SAD method).

[0107] According to one embodiment, a specified similarity measurement method used for a block matching algorithm may include, for example, a SAD method, an SSD (sum of squared differences) method, and an MSE (mean squared error) method. The SAD method may, for example, be a method of converting the difference in pixel values ​​between two blocks into an absolute value and then accumulating it for all pixels. The SSD method may, for example, be a method of squaring the difference in pixel values ​​between two blocks and then accumulating it for all pixels. The MSE method may show the average difference between blocks by calculating the average value of the SSD. For convenience of explanation, the following description uses the SAD method as an example of a specified similarity measurement method, but is not limited thereto.

[0108] Generally, the SAD method corresponds to a method of measuring similarity between two blocks using the following mathematical formula 1. The SAD method using mathematical formula 1 can be used, for example, by the motion estimation unit (320) of FIG. 3.

[0109]

[0110] Here, Pre(i,j) may mean a pixel value corresponding to position (i,j) in the reference block of the reference frame (previous frame) (e.g., "Frame N-1" in Fig. 4a).

[0111] Cur(i,j) may mean a pixel value corresponding to position (i,j) in a search block of a search frame (e.g., current frame) (e.g., "Frame N" in Fig. 4a).

[0112] m can represent the vertical size of the two blocks to be compared.

[0113] n can represent the horizontal size of the two blocks to be compared.

[0114] k may represent the index of the search block being compared with the reference block. Different indices may be assigned to each of the identified search blocks (e.g., candidate blocks) based on a specified search range (e.g., -HSR to +HSR, -VSR to +VSR), where HSR represents the horizontal search range and VSR represents the vertical search range.

[0115] According to one embodiment, a display device (100) calculates the SAD values ​​between a reference block containing an object of interest and each search block (e.g., each candidate block) within a search area using Equation 1, and identifies the search block corresponding to the index (k) having the smallest SAD value as the matching block most similar to the reference block (e.g., the block that best matches the reference block). According to embodiments, the SAD value may be referred to as a similarity measure.

[0116] However, since the SAD method using Equation 1 does not consider the shape of the object of interest, it may be difficult to accurately measure the similarity between two blocks due to factors such as background complexity, the complexity of the object of interest's shape, and noise. Therefore, the following Equation 2, which uses weight information that considers the shape of the object of interest, can be used for the SAD method. The SAD method using Equation 2 can be used, for example, by the motion estimation unit (630) of FIGS. 6a to 6c. According to the embodiments, the SAD method using Equation 2 may be referred to as an improved SAD method or a modified SAD method.

[0117]

[0118] Here, Pre(i,j) may represent the pixel value corresponding to position (i,j) in the reference block of the reference frame (previous frame) (e.g., "Frame N-1" in FIG. 4a), Cur(i,j) may represent the pixel value corresponding to position (i,j) in the search block of the search frame (e.g., current frame) (e.g., "Frame N" in FIG. 4a), wgt(i,j) may represent the weight value of the pixel corresponding to position (i,j), and the weight value may be set to a value between 0 and 1, for example; additionally, m may represent the vertical size of the two blocks to be compared, n may represent the horizontal size of the two blocks to be compared, k may represent the index of the search block being compared with the reference block, and each of the search blocks (e.g., candidate blocks) identified based on a specified search range (e.g., -HSR to +HSR, -VSR to +VSR) within a search area (e.g., search area (430) in FIG. 4a) with a different index, It can be assigned.

[0119] According to one embodiment, the display device (100) calculates the SAD values ​​between each search block (e.g., each candidate block) within a search area and a reference block containing an object of interest using Equation 2, and identifies the search block corresponding to the index (k) having the smallest SAD value as the matching block most similar to the reference block (e.g., the block that best matches the reference block). According to embodiments, the SAD value may be referred to as a similarity measure.

[0120] According to one embodiment, a display device (100) may set a weight value (wgt(i,j)) of a pixel corresponding to position (i,j) in a block (e.g., reference block, search block) as a first value based on the fact that the pixel corresponding to position (i,j) in the block is included in the region of interest of the object of interest (e.g., ball (B1) in FIG. 8) (e.g., region corresponding to the bounding box (BB1) in FIG. 8), and may set a weight value (wgt(i,j)) of the pixel corresponding to position (i,j) as a second value based on the fact that the pixel corresponding to position (i,j) is included in the background region that is not the region of interest. The first value (e.g., 1) may be set as a value higher than, for example, the second value (e.g., 0). Through this, different weight values ​​are applied to the region of interest corresponding to the object of interest (or the shape of the object of interest) and the background region that is not the region of interest, so that, for example, a higher weight may be set to the region of interest than to the background region, and similarity measurement using the SAD method may be performed considering the shape of the object of interest. Therefore, this similarity measurement method reflects the shape characteristics of the object of interest and can eliminate or reduce errors caused by the background other than the object of interest, thereby enabling accurate similarity measurement between two blocks. Through this, matching blocks containing the object of interest can be found more accurately within the search area.

[0121] According to one embodiment, the display device (100) can determine that the pixel corresponding to position (i,j) is included in the area of ​​interest of the object of interest based on the fact that the size of the overlap between the area of ​​the pixel corresponding to position (i,j) and the area of ​​interest of the object of interest is a specified size (e.g., more than half the size of the pixel area). Otherwise, it can determine that the pixel corresponding to position (i,j) is not included in the area of ​​interest of the object of interest. Through this, even when only a part of the pixel area, rather than the whole, is included in the area of ​​interest, it is possible to set an appropriate weight for the pixel by considering the exact shape of the object of interest.

[0122] According to one embodiment, the operation of identifying the shape of an object of interest of operation 510 and / or the operation of acquiring motion information of operation 520 may include an operation of performing subsampling on an image containing an object of interest. Subsampling may be performed, for example, to include the object of interest in a single block. For example, as illustrated at the bottom of FIG. 9, the object of interest (B1) may be located across a plurality of blocks (BL1a to BL1d). In this case, based on performing a block matching algorithm to acquire motion information, it may be difficult to clearly identify a reference block containing the object of interest within a reference frame (e.g., "Frame N-1" of FIG. 4a), and thus it may be difficult to perform accurate block matching. To solve this, the display device (100) may perform subsampling on the image to reduce the resolution of the image (or frame). For example, the display device (100) can convert the original image of 4K resolution (3840*2160) into an image of FHD (full high definition) resolution (1920*1080) by subsampling the original image by 1 / 4. In this case, for example, as illustrated at the top of FIG. 9, an object of interest (B1) may be included in a single block (BL1). The display device (100) can perform the block matching algorithm described above using the subsampled image. Through this, accurate block matching can be performed.

[0123] According to one embodiment, the operation of obtaining motion information of operation 520 may include the operation of obtaining a motion vector for an object of interest based on a block (matching block) of a second frame (e.g., matching block (420) of FIG. 4a) that matches a reference block (e.g., reference block (410) of FIG. 4a) containing an object of interest of a first frame. The motion vector may represent the direction and magnitude of movement of the reference block containing the object of interest of the first frame in the second frame. For example, based on the determination that the position coordinates of the reference block of the first frame are (50, 30) and the position coordinates of the matching block of the second frame that matches the reference block identified through the block matching algorithm are (55, 35), the motion vector may be V = (55 - 50, 35 - 30) = (5, 5).

[0124] In operation 530, the display device (100) may perform motion compensation based on motion information. For example, the display device (100) may add a motion compensation frame between the first frame and the second frame based on motion information. According to embodiments, the motion compensation frame may be referred to as an interpolation frame. Operation 530 may be performed, for example, by the motion interpolation unit (640) of FIGS. 6a to 6c.

[0125] According to one embodiment, as illustrated in FIG. 10, the display device (100) can generate a motion-compensated frame (e.g., an interpolated frame) (e.g., the "I(t-1)" frame of FIG. 10) between a first frame (e.g., the "I(t-1)" frame of FIG. 10) and a second frame (e.g., the "I(t+1)" frame of FIG. 10) by performing motion compensation based on motion information for an object of interest (1010). For example, the display device (100) can generate an interpolated frame between the first frame and the second frame based on motion information using a specified interpolation method. The specified interpolation method may include, for example, a linear interpolation method, a polynomial interpolation method, a non-linear interpolation method, a Gaussian interpolation method, and a nearest-neighbor interpolation method, but is not limited thereto. As an example, a linear interpolation method may be performed using the following Equation 3.

[0126]

[0127] Here, I(x,t) can mean the pixel value at position x at time t, and

[0128] v may represent a motion vector, and w may represent a weighting coefficient having a value between 0 and 1, which is used to adjust the contribution of the previous frame (e.g., "I(t-1)" frame in FIG. 10) and the next frame (e.g., "I(t+1)") in FIG. 10 during interpolation. Additionally, I(xv / 2, t-1) may represent a pixel value at position xv / 2 at time t-1, corresponding to a value calculated by reflecting the motion vector in the previous frame, and furthermore, I(x+v / 2, t+1) may represent a pixel value at position x+v / 2 at time t+1, corresponding to a value calculated by reflecting the motion vector in the next frame.

[0129] According to one embodiment, the display device (100) can interpolate the pixel values ​​of an intermediate frame (interpolated frame) (e.g., "I(t)" frame of FIG. 10) based on the temporal change between a previous frame (e.g., "I(t-1)" frame of FIG. 10) and a next frame (e.g., "I(t+1)" frame of FIG. 10) which are consecutive frames, using Equation 3. For example, the display device (100) can estimate the intermediate pixel values ​​based on the motion vector v values ​​based on the pixel values ​​of the previous frame and the next frame.

[0130] According to one embodiment, motion compensation based on motion information may be used to perform frame rate conversion (FRC), motion compensated interpolation (MIC), or motion judder cancellation (MJC). Frame rate conversion may be a technique for converting the frame rate to match an image in a display device (100) having a frame rate different from the frame rate of the original image. For example, frame rate conversion may include increasing the frame rate to match an image of 24 fps (frames per second) to a high-frame display of 60 fps. Motion compensated interpolation may be a technique for providing smoother motion by increasing the frame rate by calculating a motion vector and interpolating intermediate frames based thereon. For example, motion compensated interpolation may include converting an image of 30 fps into an image of 60 fps by adding a new frame between existing frames based on the motion vector. Motion judder cancellation is a technology that enables smooth video playback by eliminating motion judder caused by a mismatch between the frame rate and the display refresh rate.

[0131] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Expressions such as "at least one of" used herein, when placed before a list of elements, modify the entire list of elements rather than modifying each individual element of the list. For example, the expressions “at least one of A, B, and C” or “at least one of A, B, or C” can be understood as including only A, including only B, including only C, including A and B, including A and C, including B and C, or including all of A, B, and C.

[0132] Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0133] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0134] One embodiment of the present document may be implemented as software comprising one or more instructions stored in a storage medium readable by a machine. For example, the processor of the machine may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0135] According to one embodiment, the process according to the embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as memory of a server associated with the manufacturer, a server of the application store, or a relay server.

[0136] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a display device, display; At least one processor including a processing circuit; and The memory includes at least one storage medium for storing one or more instructions, and when the one or more instructions are executed by the at least one processor, the display device: Detect an object of interest included in the first frame of the image, and Based on the shape of the object of interest, motion information indicating the movement of the object of interest between the first frame and the second frame is obtained, and Based on the above motion information, a motion compensation frame is added between the first frame and the second frame, and In order to acquire the above motion information, when the above one or more instructions are executed by the above at least one processor, the display device: Further causing to set a plurality of weights including a first weight associated with a region of interest corresponding to the object of interest and a second weight associated with a background region different from the region of interest, wherein the second weight is different from the second weight, The above plurality of weights are used to select a matching block that matches a reference block containing the object of interest of the first frame among the plurality of candidate search blocks of the second frame, and The above reference block and the above matching block are a display device used to acquire the above movement information.

2. In Paragraph 1, In order to set the above different weights, when the above one or more instructions are executed by the above at least one processor, the display device: A display device that further causes a first weight value to be set on a plurality of first pixels included in the area of ​​interest and a second weight value to be set on a plurality of second pixels included in the background area, wherein the first weight value is greater than the second weight value.

3. In Paragraph 1 or 2, In order to acquire the above motion information, when the above one or more instructions are executed by the above at least one processor, the display device: Select the matching blocks using a block matching algorithm according to a specified similarity measurement scheme, and A display device that further causes to acquire the motion information based on the above reference block and the above matching block.

4. In Paragraph 3, To select the above matching block, when the above one or more instructions are executed by the above at least one processor, the display device: A display device that further causes to measure the similarity between each candidate search block of the reference block and the plurality of candidate search blocks using the specified similarity measurement method based on the first weight value and the second weight value.

5. In Paragraph 4, The aforementioned specified similarity measurement method corresponds to the SAD (sum of squared differences) method, and The above SAD method is a display device that is performed using the following mathematical formula. [Mathematical Formula] Here, Pre(i,j) means the pixel value corresponding to position (i,j) in the reference block of the first frame (denote), Cur(i,j) represents the pixel value corresponding to position (i,j) in the candidate search block among the plurality of candidate search blocks, and wgt(i,j) represents the weight value of the pixel corresponding to position (i,j), and m represents the vertical size of the above candidate search block, and n represents the horizontal size of the above candidate search block, and k represents the index of the above candidate search block.

6. In Paragraph 1, In order to identify the shape of the object of interest, when the one or more instructions are executed by the at least one processor, the display device: Using a trained artificial intelligence model, further cause to obtain information about the shape of the object of interest, and A display device in which the above-mentioned learned artificial intelligence model is trained to output output data containing information about the shape of the object of interest based on an input frame.

7. In Paragraph 6, The above output data further includes interest object detection information, and A display device comprising at least one of the above interest object detection information, type information for the type of the interest object, subtype information for the subtype of the interest object, or reliability information for the reliability level associated with the interest object.

8. In Paragraph 6, In order to identify the shape of the object of interest, when the one or more instructions are executed by the at least one processor, the display device: A display device that further causes the image to be sub-sampled at a specified rate so that the object of interest is included in one block.

9. In Paragraph 1, The first frame above is continuous with the second frame, and The above motion information corresponds to a motion vector for the object of interest, and A display device in which the above motion compensation frame is generated based on the above motion vector.

10. In Paragraph 9, The above motion compensation frame is a display device used to perform at least one of frame rate conversion, motion compensation interpolation, or motion judder cancellation.

11. A method for controlling a display device, An operation to detect an object of interest included in the first frame of an image; An operation to acquire motion information indicating the movement of the object of interest between the first frame and the second frame based on the shape of the object of interest; and Based on the above motion information, the method includes adding a motion compensation frame between the first frame and the second frame, and The action of acquiring the above movement information is: The method includes an operation of setting a plurality of weights, including a first weight associated with a region of interest corresponding to the object of interest and a second weight associated with a background region different from the region of interest, wherein the second weight is different from the second weight. The above plurality of weights are used to select a matching block that matches a reference block containing the object of interest of the first frame among the plurality of candidate search blocks of the second frame, and The above reference block and the above matching block are used in a method to obtain the above movement information.

12. In Paragraph 11, The operation of setting the above different weights is: A method comprising the operation of setting a first weight value to a plurality of first pixels included in the area of ​​interest and setting a second weight value to a plurality of second pixels included in the background area, wherein the first weight value is greater than the second weight value.

13. In Paragraph 11 or 12, The action of acquiring the above movement information is: The operation of selecting the matching block using a block matching algorithm according to a specified similarity measurement method; and A method comprising an operation to acquire the movement information based on the above reference block and the above matching block.

14. In Paragraph 13, The operation of identifying the above matching block is: A method comprising the operation of measuring the similarity between each candidate search block of the reference block and the plurality of candidate search blocks using the specified similarity measurement method based on the first weight value and the second weight value.

15. In Paragraph 14, The aforementioned specified similarity measurement method corresponds to the SAD (sum of squared differences) method, and The above SAD method is a method performed using the following mathematical formula. [Mathematical Formula] Here, Pre(i,j) represents the pixel value corresponding to position (i,j) in the reference block of the first frame, and Cur(i,j) represents the pixel value corresponding to position (i,j) in the candidate search block among the plurality of candidate search blocks, and wgt(i,j) represents the weight value of the pixel corresponding to position (i,j), and m represents the vertical size of the above candidate search block, and n represents the horizontal size of the above candidate search block, and k represents the index of the above candidate search block.

Images