Energy saving by color replacement in videos

Abstract

Apparatuses and methods are described including techniques for processing color data of a video. Techniques include obtaining an input color segment containing colors of collocated video pixels. In a first color space, a set of color segments is searched to find segments that consume less power than that consumed by the input color segment. If such a set of color segments is found, to preserve energy, the input color segment is replaced with the closest color segment from that set. If such a set of color segments is not found, in a second color space, a color segment is determined that minimizes power consumption and the distance to the input color segment. Then, to preserve energy, the input color segment is replaced with the determined color segment providing that it consumes less power than that consumed by the input color segment.

Description

[0001] ENERGY SAVING BY COLOR REPLACEMENT IN VIDEOS

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003] [1] This application claims the benefit of European Application No. 24307081.0, filed on December 12, 2024, which is incorporated herein by reference in its entirety.

[0004] BACKGROUND

[0005] [2] The increase in display resolution as well as the introduction of high dynamic range (HDR) imaging, have brought about a corresponding increase in energy consumption of display devices. Popular organic light emitting diode (OLED) displays, although relatively efficient, are a leading factor in the energy spent by content consumption in a video chain (including content production, encoding, transmission, decoding, and rendering). One way to reduce the energy consumed by a display is to manipulate the color data of the rendered video - that is, spatiotemporally modifying the colors of the video pixels without affecting the perception of those colors. However, in doing so the computational complexity should be considered as well as leveraging both low-motion and high-motion regions of the video to maximize the achieved energy reduction.

[0006] SUMMARY

[0007] [3] Aspects disclosed in the present disclosure describe methods for processing color data of a video. The methods comprise obtaining an input color segment containing colors of collocated pixels in the video. Then, in a first color space, a set of color segments is searched for, so that each segment consumes less power than that consumed by the input color segment. If such a set of color segments is found, the input color segment is replaced with a color segment (namely, a first color segment), from the set, that is the closest to the input color segment. However, if such a set of color segments is not found, in a second color space, a color segment (namely, a second color segment) is determined that minimizes both a power consumption and a distance to the input color segment. The input color segment is then replaced with the second color segment, providing that this second color segment consumes less power than that consumed by the input color segment.

[0008] [4] Aspects disclosed in the present disclosure describe apparatuses for processing color data of a video. The apparatuses comprise at least one processor and memory storing instructions. The instructions, when executed by the at least one processor, cause the apparatuses to obtain an input color segment containing colors of collocated pixels in the video. The instructions further cause the apparatuses, to search, in a first color space, for a set of color segments that each consumes less power than that consumed by the input color segment. If such a set of color segments is found, the input color segment is replaced with a color segment (namely, a first color segment), from the set, that is closest to the input color segment. However, if such a set of color segments is not found, the instructions further cause the apparatuses to determine, in a second color space, a color segment (namely, a second color segment) that minimizes both a power consumption and a distance to the input color segment. Then, the input color segment is replaced with the second color segment providing that this second color segment consumes less power than that consumed by the input color segment.

[0009] [5] Aspects disclosed in the present disclosure describe a non-transitory computer-readable medium comprising instructions executable by at least one processor to perform methods for processing color data of a video. The methods comprise obtaining an input color segment containing colors of collocated pixels in the video. Then, in a first color space, a set of color segments is searched for, so that each segment consumes less power than that consumed by the input color segment. If such a set of color segments is found, the input color segment is replaced with a color segment (namely, a first color segment), from the set, that is the closest to the input color segment. However, if such a set of color segments is not found, in a second color space, a color segment (namely, a second color segment) is determined that minimizes both a power consumption and a distance to the input color segment. The input color segment is then replaced with the second color segment, providing that this second color segment consumes less power than that consumed by the input color segment.

[0010] [6] This Summary is provided to introduce a selection of concepts in a simplified form that is further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to limitations that solve any or all disadvantages noted in any part of this disclosure.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] [7] FIG. 1 is a block diagram of an example system, according to aspects of the present disclosure.

[0013] [8] FIG. 2 is a flowchart illustrating a method for replacing colors of a video, according to aspects of the present disclosure.

[0014] [9] FIG. 3 is a flowchart illustrating a method for processing color data of a video, according to aspects of the present disclosure.

[0015] DETAILED DESCRIPTION

[0016]

[0010] Apparatuses and methods are presented herein for processing color data of a video. A system for processing, delivering, and / or displaying content, with which various aspects and examples described herein may be implemented, is generally described in reference to FIG. 1, followed by a description of the aspects of the present disclosure in reference to FIGS. 2-3.

[0017]

[0011] FIG. 1 illustrates a block diagram of an example system 100. The system 100 may be an electronic device including, for example, a personal computer, laptop computer, mobile phone, tablet computer, multimedia set-top box, digital television receiver, personal video recording system, connected home appliance, vehicle control and / or entertainment system, head-mounted display, and server. One or more elements of the system 100, singly or in combination, may be implemented as an integrated circuit (IC), multiple ICs, and / or discrete components. For example, in one embodiment, the processing, encoding and / or decoding elements of system 100 are distributed across multiple ICs and / or discrete components. In some embodiments, the system 100 is communicatively coupled to and / or in communication with other systems or devices, via, for example, a communications bus or dedicated input / output ports.

[0018]

[0012] One or more of the elements of system 100 may be provided within an integrated housing, with such elements being interconnected and able to transmit data therebetween using any suitable connection arrangement 115 generally known in the art, including, for example, an internal bus (e.g., I2C bus), wiring, and printed circuit boards.

[0019]

[0013] The system 100 includes at least one processor 110 configured to execute instructions for implementing the embodiments (or aspects) described herein, including signal (or data) coding and processing. The processor 110 may be a general -purpose processor or microprocessor, digital signal processor (DSP), one or more microprocessors in association with a DSP core, a controller, a microcontroller, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), a state machine, and the like. The processor 110 may include at least one central processing unit (CPU) and / or graphics processing unit (GPU), embedded memory, input and output interfaces, and other circuitries.

[0020]

[0014] The system 100 includes at least one memory 120, for example, a volatile memory device and / or a non-volatile memory device. The system 100 includes a storage device 140, that may be or include non-volatile memory and / or dynamic volatile memory, including EEPROM, ROM, PROM, RAM, DRAM, SRAM, DDR, flash, magnetic disk drives, solid state drives (SSD) and / or optical disk drives. The storage device 140 may be or include, for example, an internal storage device, an attached storage device, and / or a network accessible storage device. Although shown separately, the memory 120 and the storage device 140 may be collocated, integrated together, or otherwise combined.

[0021]

[0015] The system 100 includes an encoder / decoder module 130 configured to process video data and to provide encoded video data or decoded video data. The encoder / decoder module 130 may include one or more processors and / or memory (not shown). Although FIG. 1 depicts the encoder / decoder module 130 as a separate element of system 100, it will be understood that the processor 110 and the encoder / decoder module 130 may be collocated and / or integrated together as a combination of hardware and / or software, e.g., in an electronic package or chip. The encoder / decoder module 130 may be or include one or more modules that may be included in one or more separate devices that perform encoding and / or decoding functions.

[0022]

[0016] Instructions for execution by the processor 110 and / or by the encoder / decoder module 130 may be stored in the storage device 140 and subsequently loaded into memory 120 for execution by the processor 110 and / or by the encoder / decoder module 130. In some embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more items when performing the processes disclosed herein. Such items may include input video, decoded video or portions thereof, bitstreams, matrices, variables, operational logic, and intermediate and / or final results from processing of equations, formulas, or operations.

[0023]

[0017] In some embodiments, the memory of the processor 110 and / or of the encoder / decoder module 130 is used to store instructions and / or provide working memory for video encoding and decoding functions. In some embodiments, memory external to the processor 110 and / or to the encoder / decoder module 130 (e.g., the memory 120 and / or the storage device 140) is used for one or more of these functions and / or, for example, to store the operating system of a television.

[0024]

[0018] The system 100 may obtain or receive information via one or more input devices, interfaces, and / or ports as indicated in input block 105. Examples of the input devices include a radio frequency (RF) device for transmitting and / or receiving RF signals over various media, for example, RF signals received over the air from a broadcaster; component video (COMP) inputs; a Universal Serial Bus (USB) input; and / or a High-Definition Multimedia Interface (HDMI) input. Other examples include composite video input (not shown). In some embodiments, the input devices are associated with respective input processing elements, e.g., those generally known in the art. For example, the RF device may be associated with elements suitable for selecting a desired frequency (e.g., selecting or band-limiting a signal) or performing error correction on the signal. The USB and / or HDMI inputs may include respective interface processors and transceivers (or transmitters and receivers) for coupling the system 100 to other devices via USB and / or HDMI ports or connections. Various forms of input processing may be implemented, for example, by and / or within a separate input processing device or the processor 110.

[0025]

[0019] The system 100 includes a communication interface 150 that enables wired and / or wireless communication with other devices, e.g., via a communication channel 190. The communication interface 150 may include one or more transceivers, modems, network cards and the like. The communication channel 190 may be or include wired and / or wireless mediums.

[0026]

[0020] In some embodiments, data may be streamed to the system 100 via wired and / or wireless networks. Examples of such wireless networks include cellular, Bluetooth or Wi-Fi (e.g., IEEE 802.11) networks. The wired and / or wireless networks may include one or more base stations (e.g., cellular base stations, access points, etc.), and / or user equipment (e.g. cellular user equipment, stations, etc.), and / or other network elements that communicate with the system 100 via the communication interface 150 and communication channel 190, whereby the system 100 may obtain data streamed from streaming applications (e.g., OTT services) via various networks, including the Internet. In some embodiments, data is streamed to the system 100 via the input block 105 (e.g., using a set-top box that delivers data via the HDMI connection or the RF connection). In some embodiments, data is received by the system 100 in a non-streaming manner.

[0027]

[0021] The system 100 may provide one or more output signals to one or more output devices. The output devices may include a display device 165 (e.g., touchscreen display, monitor, etc.), an audio device 175 (e.g., speakers), and other peripheral devices 185, including, for example, a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. The display device 165 can be for a television, tablet, laptop, mobile phone, head-mounted display, or other device. In some embodiments, control signals are communicated between the system 100 and the display device 165, the audio device 175, and / or the peripheral devices 185, enabling device-to-device control with or without user intervention. The output devices may couple to and / or communicate with the system 100 via dedicated connections via respective display, audio, and peripheral interfaces 160, 170, 180. Alternatively, the output devices may couple to and / or communicate with the system 100 via the communication channel 190 and the communication interface 150.

[0028]

[0022] The display device 165 and the audio device 175 may be collocated, integrated, or otherwise combined with the other components of system 100 in a single unit (e.g., a television). Alternatively, the display device 165 and the audio device 175 may be separate from one or more of the other components of the system 100. In embodiments in which the display device 165 and the audio device 175 are external components, the output signals may be provided via dedicated outputs and / or connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0029]

[0023] The increase in display resolution (from SD to HD to 4K and soon to 8K and beyond) and the introduction of HDR image rendering have brought about a corresponding increase in energy consumption of display devices. Displaying content is the most energy consuming process in the video chain, in both battery-powered devices (e.g., smartphones) and grid-connected devices (e.g., a TV). Recently, OLED displays have become popular because of numerous advantages they have over non-emissive displays such as Thin-Film Transistor Liquid Crystal Displays (TFT-LCDs). Rather than using a uniform backlight, OLED displays are composed of LEDs whose emitted lights represent the rendered pixels of the displayed image content. The power consumption of OLED displays is therefore highly correlated with image content and can be directly estimated by considering the luminance level of the rendered pixels. Although OLED displays consume energy in a more controllable and efficient manner, employing them is still the most energy consuming process in the video chain.

[0030]

[0024] One approach to control the energy consumption of a display is by manipulating the color data of the rendered images. Hence, colors of (all or selected) image pixels may be modified, across time and / or across space, without affecting the perception of those colors. The modified colors are perceived as the original colors, but their rendering on the display consumes less energy. For example, to display an image on an OLED display, the image is rendered by generating light pulses that are emitted from respective pixel positions. The energy consumed by the generation of these light pulses can be reduced by modifying the color values of the respective pixels. This can be done while maintaining the perceived quality of the rendered image. For example, original colors may be replaced by pairs of colors, either alternating spatially or temporarily, which reproduce the original colors’ stimuli while consuming less energy.

[0031]

[0025] Various aspects of this approach are implemented by algorithms described, for example, in: “method and device for pixel color replacement in video based on complementary colors” (EP application no. 23306641.4, filed September 29, 2023); “quality constrained method and device for energy reduction control of visual content using alternating complementary colors” (EP application no. 23305865.0, filed June 01, 2023); “method and device for adjustable energy reduction control of visual content using alternating complementary colors” (EP application 23305836.1, may 26, 2023); “method and device for energy reduction control of visual content” (EP application no. 23305480.8, filed April 03, 2023); “method and device for reducing display energy by using spatially alternating complementary colors” (EP application no. 22306997.2, filed December 22, 2022); “method and device for reducing flicker for successive pixels of temporally alternating complementary colors” (EP application no. 22306994.9, December 22, 2022); and “method and device for reducing display energy by using temporally alternating complementary colors” (EP application no. 22306995.6, filed December 22, 2022).

[0032]

[0026] To reduce the energy consumed by rendering of video content while not impairing the perceived quality of that content, the video colors can be replaced with other colors. To that end, techniques (such as those described in EP application no. 22306995.6, 22306994.9, or 22306997.2) are applied independently to each frame in a video to replace the color of each pixel with a pair of colors (e.g., replacing each frame with two frames). However, in such an approach, there is no accounting for video coherency - that is, the changes in color that a video pixel is experiencing overtime. Furthermore, in the techniques mentioned above (where the color of one pixel is replaced by a pair of colors) to reach an optimal power gain, the resulted pair of colors may be each perceptually very different from the color they are replacing (see, EP application no. 22306995.6, 22306994.9, or 22306997.2). Although this difference can be decreased (see EP application no. 22306997.2), such a decrease comes at the expense of the gain in energy reduction.

[0033]

[0027] In EP application no. 23306641.4, a technique is proposed to achieve color similarity between a given color pair (of collocated pixels in successive frames) and the replacing color pair. This is done by, first, searching for the locus of minimal power in some color space (that is, the set of replacing color pairs that minimize the power). Next, a color pair that also minimizes the distance to the given color pair is selected from that set. However, in this technique, the computations are done in the XYZ color space, which enables the finding of a set of replacing color pairs, where each pair is of colors that are very different from each other. This technique is, thereby, limited to cases where the given color pair is of collocated pixels from a high motion region of the video. Thus, a goal of aspects described herein may be replacing the colors of collocated pixels, from both high-motion and low-motion video regions, with respective similar colors whose usage (e.g., rendering) consumes less energy.

[0034]

[0028] When energy reduction is achieved by replacing the color of a pixel by another single color, the used three dimensional searching space (spanned by the three color components of the pixel) may be limiting. In contrast, when the color of a pixel is replaced by a pair of colors, the searching space may reach six dimensions, and so the likelihood of finding a replacing pair of colors that provides a larger gain in energy reduction increases. The techniques described in EP application no. 22306995.6, 22306994.9, and 22306997.2, for example, take advantage of the six dimensional searching space when replacing the color of each pixel with a pair of colors. However, in the case of videos, instead of replacing the color of each pixel, it is further possible to replace the colors of two (or more) collocated pixels (e.g., in successive frames) by another two (or more) colors. In this case, the dimension of the search space may be further increased, as the two (or more) replacing colors can be searched in a six dimensional space (or a larger space) and that may produce higher energy savings.

[0035]

[0029] Aspects disclosed in the present disclosure provide color segments (e.g., color pairs) for replacing respective original color segments (e.g., original color pairs) of collocated pixels from (e.g., successive) video frames. The replacing colors are each close to the original colors; and their usage consumes less energy compared to the original colors they are replacing. This may be achieved by a multiple-stage process. Generally, in a first stage, colors of video pixels are processed, in a given color space, to obtain replacing colors, using a power constraint. Thus, for a given color segment (e.g., a color pair), a set of replacing color segments (e.g., color pairs) is found in which each segment minimizes the power consumption. Then, one color segment is selected from that set that also minimizes a distance metric. In a second stage, for a given color segment for which no replacing color segment could be found in the first stage, a replacing color segment is searched, in another color space, using both a power constraint and a distance constraint. This process allows both 1) gaining energy reduction when replacing colors of pixels from static regions as well as from dynamic regions of the video frames, and 2) finding replacing colors that are each perceptually close to the respective colors they are replacing.

[0036]

[0030] Previously disclosed techniques are described next. Notice that a color represented in the RGB color space is denoted herein by CRGBand a corresponding color represented in another color space, for simplicity of the presentation, is denoted herein by C (e.g., based on the context, C = CLabor C = CXYZ).

[0037]

[0031] An alternating complementary colors (ACC) algorithm is described below as an example of a technique for replacing the colors of pixels of an image with respective pairs of colors with the goal of reducing the energy consumed by the rendering of the image on a display. Further description of the ACC algorithm can be found in EP Application No. 22306995.6.

[0038]

[0032] Thus, for a pixel of an image having a color value in an RGB color space, that is, an input (original) color CRGB= R0G0B0, the following steps are carried out to find a replacing pair of colors.

[0039] A uniform color space is selected, having a gamut denoted by G. When a Lab color model is used (as demonstrated herein), a forward color transforms TT(RGB) can be used to convert an RGB value to a corresponding value in the (CIE)XYZ color space and then to a corresponding value in the Lab color space (i.e., RGB- XYZ- Lab). Alternatively, the conversion may be from the RGB color space directly to the Lab color space (i.e., RGB- Lab). An inverse color transforms JT Lab) can be used to apply inverse operations (i.e., Lab- XYZ- RGB or Lab -> RGB).

[0040] The input color CRGB= R0G0B0is transformed to a corresponding color Coin the selected uniform color space. For example, C0≡ C0Lab= L0a0b0= ℱ풯(R0G0B0).

[0041] The uniform color space is sampled within the gamut G to get a series of N candidate colors, such that a candidate color Ci= Liaibisatisfies a given criteria. For example: 1) at= bt= 0 so that is part of the grey ramp; 2) the cylindrical version Licihihas the same luminance and hue as L0a0b0but maximum chroma ci; 3) L; = Loto avoid any variation in luminance; or 4) Lt= a ■ Lowith a E [0.8, 1.2] to limit variation in luminance (quasi- equiluminance).

[0042] For each of the N candidate colors,. the following steps are performed:

[0043] a color Cj= Ljajbjrelated by symmetry to Ci= Liaibiis obtained, as follows: Lj= 2 · L0− Li

[0044] aj= 2 · a0− ai

[0045] bj= 2 · b0− bi

[0046] - It is verified that Cjis within the boundaries of the gamut G. If not, the color sample Ci is excluded from the series of candidate colors.

[0047] - The color pair Ciand Cjare transformed from the uniform color space to the RGB color space:

[0048] CiRGB= RiGiBi= 풥풯(Liaibi), and

[0049] CjRGB= RjGjBj= 풥풯(Ljajbj).

[0050] - The energy (or power) consumed by the pair CRGBand CRGBif rendered on a display, each for half a frame period, is evaluated as follows:

[0051] 풫ij= 풫½(CiRGB) + 풫½(CjRGB).

[0052]

[0053] For i = 1: N, a color pair (CRGBand CRGB) that yields a minimum powermin(the minimum value obtained for

[0054]

[0055] ) is identified. If that minimum power is smaller than the power consumed by the rendering of the input color CRGB, that is, J’mm < P " o(CR GB), then the identified color pair is used to replace the input color CRGBotherwise the input color is not replaced.

[0056]

[0033] An algorithm for replacing complementary colors in video, namely, VRCC, is described below as another example of a technique for replacing colors. In this case, colors of pairs of collocated pixels from successive video images are replaced with respective pairs of colors with the goal of reducing the energy consumed by the rendering of the video images on a display. Further description of the VRCC algorithm can be found in EP Application No. 23306641.4.

[0057]

[0034] The colors, in an RGB color space, of a pair of collocated pixels from successive video images, and Im2, are respectively denoted by C1RGB= R1G1B1and C2RGB= R2G2B2. Thus, the following steps are carried out for an input (original) color pair, CRGBand CRGB, to find a replacing color pair.

[0058] A color space is selected, having a gamut denoted by G. When an XYZ color space is used (as demonstrated herein), a forward color transforms ℱ풯(RGB) can be used to convert an RGB value to a corresponding value in the (CIE)XYZ color space (i.e., RGB- XYZ) and an inverse color transforms T(XYZ) can be used to apply the inverse operation (i.e., XYZ- RGB).

[0059] The input color pair,

[0060]

[0061] C1RGB= R1G1B1and C2RGB= R2G2B2, is transformed to a corresponding color pair in the selected color space, XYZ - that is, ≡ C1XYZ= X1Y1Z1= 풻풯(R1G1B1) and C2≡ C2XYZ= X2Y2Z2= 풻풯(R1G1B1).

[0062] The color space is sampled within the gamut G to get a series of N candidate color pairs. Each candidate color pair, denoted = XiYiZiand = XJYJZJ. satisfies:

[0063] Xj= X1+ X2− Xi

[0064] Yj= Y1+ Y2− Yi

[0065] Zj= Z1+ Z2− Zi

[0066] For each of the N candidate color pairs, and, the following steps are performed:

[0067] - The color pair and are transformed from the XYZ color space to the RGB color space:

[0068] CiRGB= RiGiBi= 풥풯(XiYiZi), and

[0069] CjRGB= RjGjBj= 풥풯(XjYjZj).

[0070] - The energy (or power) consumed by the pair CBGBand CRGBif rendered on a display is evaluated as follows:

[0071]

[0072] = 풫(CiRGB) + 풫(CjRGB)

[0073] For i = 1: N, those pairs that yield a minimum power

[0074]

[0075] (the minimum value obtained for Pij) are identified. The subset of pairs, and (corresponding to the pairs of CRGBand CRGB), that yield the minimum power identify the locus ' / Zj in the XYZ color space. Thus,

[0076]

[0077] if J’mm > ^(Cf*75) + P(C2GB\ the input color pair, Ci and C2, is not replaced. But, if ^min < (GRGB) + P(C2GB- a color pair, and, is selected from the subset. The selected color pair minimizes the distance with the pair Ci and C2.

[0078]

[0035] As mentioned before, applying VRCC to a video successfully results in power reduction. However, such success is mostly limited to high motion regions of the video (where colors largely change overtime). However, wide-spectrum of videos also contain low motion regions. Aspects described herein, address both low-motion and high-motion regions of the video. In doing so, in contrast to ACC, the colors produced are perceptually similar to the colors they are replacing; and, in contrast to VRCC, both low motion and high motion regions are considered, thus potentially enabling higher gain in energy savings. To that end, aspects described herein operate in two color spaces: a first color space that is less computationally expensive (e.g., a linear color space such as XYZ) and a second color space that is more computationally demanding (e.g., a uniform color space such as Lab) - where the latter can be used to target low-motion regions that where not addressed by the former.

[0079]

[0036] FIG. 2 is a flowchart illustrating a method for replacing colors of a video 200, according to aspects of the present disclosure. The method 200 is employed to replace colors of collocated pixels from (e.g., successive) video frames. Although the method is demonstrated with respect to replacing the colors of two collocated pixels, the colors of three or more collocated pixels (i.e., a color segment) may be replaced using the principles described herein. As illustrated in FIG. 2, an input color pair 210 - that is, original colors of two collocated pixels from respective video frames - may be processed in two color spaces with the objective of finding a replacing color pair that consumes less energy when used (e.g., rendered), as described below.

[0080]

[0037] In step 220, in a first color space, a set of color pairs may be searched that each consumes a minimal powerminwhen used. This set of colors resides on a locus of minimum power in the color space used. The input color pair may be represented in an RGB color space, and the first color space may be an XYZ color space (having a better linearity characteristics that benefit computations in that space). Thus, according to aspects, the following processing may be carried out in this step 220 to find a set of color pairs. First, the color space may be searched for color pair candidates with respective mean values that are within a range of (or the same as) the mean value of the input color pair to maintain the same color appearance. Second, the power consumed by the usage of each of the color pair candidates may be computed. Then, those color pair candidates that correspond to a power consumption of minimum value form the set of color pairs. Thus, in an aspect, the set of color pairs may be generated by searching the color space for color pairs, so that 1) the mean value of each color pair, and, is the same as (or within range of) the mean value of the input color pair and C2. and 2) the power consumed by each color pair is minimal, 풫ij= 풫min.

[0038] Next in step 230, it is checked whether the color pairs in the set (if not empty) correspond to a minimal powerminthat is less than that consumed by the input color pair -that is, it is checked whether ’mm < ^(C *75) + ^(C *75). If so, in step 240, the set is searched for a color pair, and, that is the closest to the input color pair, in order to achieve a color pair, and, that is perceptively the same as the input color pair it is replacing. The found color pair is used to replace the input color pair, in step 280. However, if the color pair set (to be generated in step 220) is empty, or, otherwise, if the color pairs in the set correspond to a minimal power that is not less than that consumed by the input color pair - that is,min>

[0081]

[0082] P(CfGB) + J’(C2RCB) - then processing proceeds to step 250.

[0083]

[0039] In an aspect, steps 220, 230, and 240 are performed for input colors of respective collocated pixels. The pixels for which replacing colors where not found (a negative answer for the query in step 230) may be recorded by a mask. Such a mask can be then used by step 250 to determine the pixels whose colors should be further processed. In this way, the processing in step 250 can be restricted to colors of pixels for which adequate results were not obtained by previous steps.

[0084]

[0040] In step 250, in a second color space, a pair of color, and, that minimizes both power consumption and distance to the input color pair 210 may be searched. In an aspect, the used color space may be a uniform color space such as Lab. Thus, according to aspects, the following processing may be carried out in step 250. First, the color space is searched for color pairs having a mean value that is the same as (or within a range to) the mean value of the input color pair 210. This is done in order to maintain perceptual similarity with the input color pair to be replaced. Then, in a first aspect, out of the color pairs found, a color pair is further searched for that minimizes 1) a power consumption, J’y =

[0085]

[0086] J’(Q) + J’(Q), and then 2) a distance to the input color pair, A(C, ) (or, equivalently, 21 (C2, )) in the color space. In an alternative of the first aspect, out of the color pairs found, a color pair is further searched for that minimizes 1) a distance to the input color pair, Δ(C1, Ci) in the color space, and then 2) a power consumption, 풫ij. In a second aspect, out of the color pairs found, a color pair is further searched for that minimizes a sum of 1) the power consumption, 풫ijand 2) the distance to the input color pair, Δ(C1, Ci). Note that minimizing the sum of 풫ijand Δ(C1, Ci) is less restrictive than minimizing

[0087]

[0088] and Δ(C1, Ci).

[0089]

[0041] For example, in a further aspect of step 250, the color space is searched for color pairs having a mean value that is the same as (or within a range to) the mean value of the input color pair 210. Then, following the first aspect, out of the color pairs found, a first set of color pairs may be further searched for that minimize 풫ijand Δ(C1, Ci). If the searched for first set contains more than one color pair, than, following the second aspect, out of the first set of color pairs, a second set of color pairs may be further searched for that minimizes the sum of 풫ijand Δ(C1, Ci). Notice that the minimization process (carried out to find the first set or the second set) may be stopped when the first color pair is found; or, if more than one color pair is found, another criterion may be used to select one of them (e.g., picking one at random or selecting the one that is within a closer range to the input color pair 210). The color pair(s) found in the first set or (if searched for) in the second set are further considered in step 260.

[0090]

[0042] Referring back to FIG. 2, if a color pair that minimizes 풫ijand Δ(C0, Ci) (or that minimizes the sum of 풫ijand Δ(C0, Ci)) is found, it is determined, in step 260, whether the found color pair corresponds to a power consumption 7Q that is lower than that consumed by the input color pair, this is, whether 7Q < J’( ) + ^(Q)- If so, the found color pair, for whichP " ij < P " Ci) + J’(Q), is used t° replace the input color pair,

[0091]

[0092] and C2, in step 280. Otherwise, if J’y > J’( ) + J’(C2), then the input color is kept, in step 270 (i.e., the input color 210 is not replaced).

[0093]

[0043] In an aspect, Lagrangian-based optimization may be employed (in step 250) to find a color pair that minimizes power consumption and the distance to the input color pair. In this aspect, a Lagrangian multiplier may be formulated as follows:

[0094] C

[0095]

[0096] OST = a(^(Ct) + J’(Q)) + A (d(C’1, Q)).

[0097] The scaling parameter a, used to normalize the power term, is directly related to the power gain range. A higher value of a can be used for lower levels of power (i.e.,

[0098]

[0099] ZP (Q) + J’(Q))) and a lower value of a can be used for higher levels of power. The parameter A can be determined based on Δ using an initial value for the color pair, Ciand Cj, and may then be dynamically adjusted during optimization, based on intermediate results for the color pair, to obtain a final color pair that is closer to the input color pair C1and C2.

[0100]

[0044] In an aspect, the distance function 21 can be a norm, Δ = ||C1− Ci|| (or, equivalently, Δ = ||C2− Cj||). In another aspect, the distance function can be based on a cosine similarity, Δ = (1 − cosine(C1, Ci)) (or, equivalently, Δ = (1 − cosine(C2, Cj))). Alternatively, the distance function can be a sum of both a norm and a cosine similarity, as follows:

[0101] COST = a (tP(Q) + J’(Q)) + A (||Q - Q|| + (1 — cosine(Q, ))).

[0102]

[0103] Alternatively, two distance functions may be used as follows:

[0104] COST = a (tP(Q) + J’(Q)) + ||Q - Q|| + A2(l — cosine(Q, 6)}).

[0105]

[0106] In this case, the λ1and the λ2parameters may be determined dynamically based on the evolution of the distance functions during optimization. In an aspect, the power term and two distance terms may be optimized altogether with the dynamic adjustment of λ1and λ2.

[0045] FIG. 3 is a flowchart illustrating a method for processing color data of a video 300, according to aspects of the present disclosure. The method may begin, in step 310, with obtaining an input color segment, containing colors of collocated pixels in the video. The input color segment may contain colors of two or more collocated pixels from (e.g., successive) frames of the video (e.g., a color pair as described with respect to FIG. 2). Next, in steps 320 to 350, color data is processed with the objective of finding a similar color segment that, when used, consumes less power than the power consumed by the input color segment. Processing may be performed in multiple color spaces involving different computational complexity. For example, a first color space (e.g., used by the processing of steps 320-330) may be a linear color space such as the XYZ color space, and a second color space (e.g., used by the processing of steps 340-350) may be a uniform color space such as the Lab color space. By limiting the amount of processing in the uniform color space the computational complexity may be decreased.

[0107]

[0046] In step 320, in a first color space, a set of color segments may be searched for, so that each of the color segments in the set consumes less power than the power consumed by the input color segment. In an aspect, color segments may be searched for so that each consumes a minimum power consumption that is lower than the power consumed by the input color segment. To maintain perceptual color consistency, the searched color segments may be limited to color segments with respective mean values that are within a range of (or the same as) a mean value of the input color segment. If the searched for set of color segments is found (in step 320), then, in step 330, the input color segment may be replaced with a color segment (namely, a first color segment) from the set that is the closest to the input color segment. However, if the searched for set of color segments is not found (in step 320), processing proceeds, in steps 340 and 350, in a second color space as described below.

[0108]

[0047] In step 340, a segment color (namely, a second color segment) may be determined that minimizes both power consumption and the distance to the input color segment (e.g., using a distance metric based on cosine similarity). To maintain perceptual color consistency, the second color segment may be limited to a color segment with a mean value that is within a range of (or the same as) a mean value of the input color segment. Then, in step 350, if that second color segment consumes less power than the power consumed by the input color segment, the input color segment may be replaced with the second color segment. In the case that the second color segment does not consume less power than the power consumed by the input color segment, then the input color segment is not replaced.

[0109]

[0048] The first color segment or the second color segment, as well as the input color segment they may be replacing, may contain any number of colors. In an aspect, the first color segment or the second color segment may contain a number of colors that is the same as the number of colors in the input color segment (e.g., a pair of colors). In another aspect, the first color segment or the second color segment may contain a number of colors that is larger than the number of colors in the input color segment (e.g., more than two colors). Using a color segment with a large number of colors increases the number of color combinations to choose from, thereby, the achievable power gain may increase.

[0110]

[0049] One or more embodiments provide a computer program comprising instructions which when executed by one or more processors cause such processors to perform the methods according to any of the embodiments described above. One or more embodiments also provide a computer readable storage medium having stored thereon instructions for performing the methods according to any of the embodiments described above.

[0111]

[0050] One or more embodiments provide a computer readable storage medium having stored thereon video data generated according to the methods described above. One or more embodiments also provide a method and apparatus for transmitting or receiving video data generated according to the methods described above.

[0112]

[0051] The embodiments described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., as a method), the implementation of such features may also be implemented in other forms. An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. Corresponding methods may be implemented in, for example, a processor.

[0113]

[0052] Various numeric values are used in the present application. Such specific values are for example purposes and the embodiments described are not limited to these specific values.

[0114]

[0053] Various methods are described herein, and such methods comprise one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., for example, a “first operation” and a “second operation”. Use of such terms does not imply an order to the operations unless specifically required.

[0115]

[0054] The present disclosure may refer to “determining” various pieces of information. Determining information may include one or more of, for example, estimating, calculating, predicting, or retrieving (e.g., from memory) the information.

[0116]

[0055] The present disclosure may refer to “accessing” various pieces of information. Accessing information may include one or more of, for example, receiving, retrieving (e.g., from memory), storing, moving, copying, calculating, determining, predicting, or estimating the information. Similarly, the present disclosure may refer to “receiving” various pieces of information. Receiving information may include one or more of, for example, accessing or retrieving (e.g., from memory) the information.

[0117]

[0056] It is to be understood that use of any of the following

[0118]

[0119] “and / or”, and “at least one of’ is intended to encompass all possible selections of listed items, taken either individually or in any combination thereof.

[0120]

[0057] While specific embodiments have been described in the foregoing description in connection with the accompanying drawings, it should be understood that embodiments described herein are examples only and should not be taken as limiting the scope of the present disclosure or the following claims. Although features and elements are described herein in particular combinations, those of ordinary skill in the art will appreciate that such features or elements may be used alone or in any combination with the other features and elements. It is understood, therefore, that the overall teachings of the present disclosure are not limited to the particular embodiments, implementations, and examples disclosed herein, but are intended to cover variations, modifications, and alternatives as defined by the appended claims and any and all equivalents thereof.

Claims

1. CLAIMS1. A method for processing color data of a video, comprising:3.obtaining an input color segment containing colors of collocated pixels in the video; in a first color space, searching for a set of color segments that each consumes less power than that consumed by the input color segment;4.if the searched for set of color segments is found, replacing the input color segment with a first color segment, from the set, that is closest to the input color segment; and5.if the searched for set of color segments is not found, in a second color space, determining a second color segment that minimizes a power consumption and a distance to the input color segment, and6.if the second color segment consumes less power than that consumed by the input color segment, replacing the input color segment with the second color segment.

2. An apparatus for processing color data of a video, comprising:8.at least one processor; and9.memory storing instructions that, when executed by the at least one processor, cause the apparatus to:10.obtain an input color segment containing colors of collocated pixels in the video, in a first color space, search for a set of color segments that each consumes less power than that consumed by the input color segment,11.if the searched for set of color segments is found, replace the input color segment with a first color segment, from the set, that is closest to the input color segment, and if the searched for set of color segments is not found, in a second color space, determine a second color segment that minimizes a power consumption and a distance to the input color segment, and12.if the second color segment consumes less power than that consumed by the input color segment, replace the input color segment with the second color segment.

3. The method according to claim 1, or the apparatus according to claims 2, wherein the searching for a set of color segments comprises:searching for a set of color segments that each consumes a minimum power consumption, wherein the minimum power consumption is lower than a power consumed by the input color segment.

4. The method according to claim 1 or 3, or the apparatus according to claim 2 or 3, wherein the searching for a set of color segments comprises:15.searching for a set of color segments that each has a mean value that is within a range of a mean value of the input color segment.

5. The method according to claim 1 or 3, or the apparatus according to claim 2 or 3, wherein the searching for a set of color segments comprises:17.searching for a set of color segments that each has a mean value that is the same as a mean value of the input color segment.

6. The method according to claim 1 or 5, or the apparatus according to claim 2 or 5, wherein the determining of a second color segment comprises:19.determine a second color segment having a mean value that is within a range of a mean value of the input color segment.

7. The method according to claim 1 or 5, or the apparatus according to claim 2 or 5, wherein the determining of a second color segment comprises:21.determine a second color segment having a mean value that is the same as a mean value of the input color segment.

8. The method according to any one of claims 1 or 3-7, or the apparatus according to any one of claims 2-7, wherein the input color segment contains a pair of colors.

9. The method according to any one of claims 1 or 3-8, or the apparatus according to any one of claims 2-8, wherein the first color segment or the second color segment contains a number of colors that is different than the number of colors contained in the input color segment.

10. The method according to any one of claims 1 or 3-9, or the apparatus according to any one of claims 2-9, wherein the distance to the input color segment is a distance metric based on a cosine similarity.

11. The method according to any one of claims 1 or 3-10, or the apparatus according to any one of claims 2-10, wherein the first color space is a linear color space.

12. The method according to any one of claims 1 or 3-11, or the apparatus according to any one of claims 2-11, wherein the second color space is a uniform color space.

13. The method according to any one of claims 1 or 3-12, or the apparatus according to any one of claims 2-12, wherein the determining of a second color segment in the second color space comprises:27.determining a second color segment that minimizes a sum of a power consumption and a distance to the input color segment.

14. The method according to any one of claims 1 or 3-12, or the apparatus according to any one of claims 2-12, wherein the determining of a second color segment in the second color space comprises:29.searching for a first set of color segments that minimize a power consumption and a distance to the input color segment;30.if the searched for first set contains more than one color segment, searching for a second set of color segments that minimize a sum of a power consumption and a distance to the input color segment; and31.selecting a second color segment from the second set of color segments.

15. The method according to any one of claims 1 or 3-14, or the apparatus according to any one of claims 2-14, wherein the collocated pixels in the video are from successive frames of the video.

Images